By Gokalp Katkat & Ege Sabanci
LLM Data Sovereignty: Why Enterprises Keep Models In-House
TL;DR
Data sovereignty is the reason most regulated enterprises cannot use hosted LLM APIs: the moment proprietary data crosses a network boundary, control over it is shared. This guide explains what data sovereignty means for AI, why it is driving on-prem adoption, and how to evaluate a deployment against your sovereignty requirements.
Data sovereignty is the quiet driver of on-prem AI
What is LLM data sovereignty? It is the legal and operational control that keeps your models and proprietary data inside your boundary, under your jurisdiction, keys, and governance, with no external processing or dependency.
Most discussions of on-prem LLMs focus on cost and latency. But the reason regulated enterprises actually move models in-house is simpler and more fundamental: data sovereignty. When proprietary data crosses a network boundary to a hosted API, control over that data is no longer exclusively yours — no matter what the contract says.
This guide explains what data sovereignty means for AI, why it is reshaping how enterprises buy models, and how to evaluate a deployment against your sovereignty requirements.
The shift is already visible in procurement. Privacy reviews that once asked how a provider stores prompts now ask whether prompts leave the boundary at all. That single change — from where data is stored to whether it leaves — is what moves a workload from hosted processing to a self-contained deployment, and it is the standard this guide uses throughout.
What data sovereignty means for AI
Data sovereignty is the principle that data is subject to the laws and governance of the jurisdiction where it resides — and, for an enterprise, that the organization retains full control over its own data. Applied to AI, it means the data you feed a model, and the model itself, stay under your control.
A hosted API breaks this in a subtle way. Even if the provider commits not to train on your data, your data is processed on their infrastructure, in their jurisdiction, behind their access controls. For a bank, an insurer, or a healthcare provider, that is a compliance exposure they cannot accept.
In operational terms, sovereignty adds three hard constraints: the model runs where your data lives, the data plane never requires an external call to produce a result, and every access is logged under your controls. Anything that violates one of those constraints is a boundary crossing, not a configuration detail, and must be justified against the same standard as moving a database outside the jurisdiction.
Why it is driving on-prem adoption
The consequence is that the most sensitive AI workloads cannot use hosted APIs at all. The only way to keep data sovereign is to run the model where the data lives — on infrastructure you own, behind your network, with your keys.
That is the shift: enterprises are not choosing on-prem because it is cheaper (though it often is at volume). They are choosing it because it is the only architecture that satisfies their data boundary. Sovereignty is a hard requirement, and on-prem is the answer.
The boundary is also becoming more practical to hold. Our two-stage REAP + AWQ pipeline on LFM2.5-8B-A1B (2026-07-20) prunes the MoE from 32→16 experts across 22 MoE layers and then packs the survivors to W4A16_ASYM group 128, yielding a 2.79 GB packed INT4 artifact that runs without an external dependency — full method and run summaries in REAP + AWQ MoE compression. Smaller artifacts make a strictly in-boundary deployment easier to size, ship, and audit without weakening the control.
Evaluating a deployment against sovereignty requirements
When you evaluate an on-prem model, ask these questions:
- Where does the model run? On your hardware, or on the vendor's? If the vendor runs serving, your data plane is not yours. For self-hosted families that run directly on your VPC or bare metal, see model families for in-house deployment.
- Does the model phone home? Telemetry, licence checks, and update calls are all data leaving your boundary. A sovereign deployment must be self-contained. If you need a boundary-preserving integration designed around your controls, see custom solutions.
- Can you run it air-gap? If your requirements are strict, the model must run with no external dependency at all — no download at inference, no licence callback, no remote config.
- Who holds the keys? You should control the weights, the serving stack, and the access. Compare ownership, licence, and support terms on pricing for licences and managed inference.
A yes on every question is the minimum for a defensible posture. Any no is not a trade-off to document — it is a boundary exception that must be approved, logged, and periodically re-justified against the same standard you apply to data stores.
Sovereignty is an architecture, not a feature
Data sovereignty is not something you toggle on. It is a property of the architecture: where the model runs, where the data lives, and who controls the plane between them. If those decisions are made with sovereignty in mind from the start, the deployment is defensible. If they are bolted on later, they are not.
For enterprises in regulated industries, that is the real reason to run models in-house — and the standard against which any on-prem offering should be judged.
Treat the evaluation as an architecture review, not a feature checklist. Verify the runtime has no external dependency, that weights and caches never leave your network, and that audit covers every access under keys you hold. When the architecture is sovereign, compliance, privacy, and long-term ownership follow; when it is not, no contractual clause restores the control you ceded.
Sovereignty checklist: who controls what
Use the checklist below to compare a self-contained deployment against a hosted API on the controls that matter for sovereignty. Each row is a question your audit will ask.
| Control | In-house | API |
|---|---|---|
| Data residency | Data and model stay in your jurisdiction and network; no external processing | Prompts and context processed on provider infrastructure in provider jurisdiction |
| Keys | You hold model, cache, and access keys; revocation is yours | Provider holds serving and access keys; you trust their IAM |
| Audit | You own logs; every access is under your SIEM and retention | Audit is provider-side; you receive filtered exports |
| Egress | No egress at inference; air-gap capable; firewall can deny all outbound | Network egress per request; telemetry and billing callbacks required |
Sovereignty controls: in-house versus API. A yes in the in-house column is the expected answer for a defensible deployment.
Verify the boundary: audit and egress checks
A sovereign claim must be verifiable. The snippets below check that no data leaves the boundary at inference and that every access is logged under your controls.
# 1) check the sovereign audit log for any egress
curl -s http://localhost:8000/v1/audit/logs | jq '.events[] | select(.egress != null)'
# expect: no output (no egress events at inference)
# 2) firewall rule: deny and log any outbound from the model user
iptables -A OUTPUT -m owner --uid-owner konic-llm -j LOG --log-prefix "egress-blocked: " --log-level 4
iptables -A OUTPUT -m owner --uid-owner konic-llm -j DROP
# 3) verify air-gap: run inference with external network down
sudo iptables -A OUTPUT -j DROP && curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"konic-lfm2.5-8b-reap-awq","messages":[{"role":"user","content":"hello"}]}' | jq . && sudo iptables -D OUTPUT -j DROP
# healthy sovereign deployment: inference succeeds with no external dependencyRun these checks in CI for every release. If either the audit shows an egress event or inference fails without network, the deployment is not sovereign — fix the runtime before you certify the boundary.
Frequently asked questions
Three questions teams ask when they evaluate a model against sovereignty requirements.
How is sovereignty different from residency?
Residency is where data is stored; sovereignty is who controls the entire plane — storage, processing, keys, and audit — and under which jurisdiction. A workload can be resident in your region but still processed on a provider's infrastructure behind their controls. Sovereignty requires that processing also stays inside your boundary, with logs and keys you hold, so no external party can access or move the data without your explicit action.
What does GDPR require for model data?
GDPR requires a lawful basis, purpose limitation, and appropriate safeguards for transfers — and, for sensitive workloads, data minimization at processing. Sending prompts that contain personal data to a hosted API is a transfer that needs its own basis and safeguards, even if the provider promises not to train. Keeping the model in-house, with prompts never leaving your boundary and audit under your retention, removes the transfer entirely and leaves you with the familiar obligations you already apply to your data stores.
Does keeping models in-house create lock-in?
Only if you let it. In-house with open weights, your serving stack, and keys you control is the opposite of lock-in — you can change vendors, runtimes, or hardware without migrating data out of a provider. Lock-in comes from managed endpoints that keep the weights and the serving behind their API, or licences that require a callback to function. Evaluate ownership on model families and licence terms so the model remains an asset you hold, not a service you rent.
Author
Written by Gokalp Katkat and Ege Sabanci — founders and research team at Konic Labs, Inc. Open, reproducible research for on-prem enterprise LLMs.
More guides
Air-Gapped LLM Deployment: What It Means and When You Need It
An air-gapped LLM runs on infrastructure with no connection to the public internet — the model, the data, and the serving stack all live inside your boundary. This guide explains what air-gapped deployment actually requires, which industries need it, and the practical constraints of running a model with no external dependencies.
How to Deploy an LLM On-Premise: A Practical Guide
Deploying an LLM on your own infrastructure is a sequence of concrete decisions: pick the model, size the hardware, choose a serving runtime, quantize for your GPU, and wire in monitoring. This guide walks each step with the trade-offs that actually matter for a production on-prem deployment.
On-Prem LLM vs API: When to Self-Host Your Language Models
API-based LLMs are fast to adopt but scale poorly on cost, latency, and data control. On-prem LLMs trade setup effort for predictable economics, lower per-token cost at volume, and data that never leaves your boundary. This guide breaks down the decision across cost, latency, privacy, compliance, and operational effort — and when each path is the right call.
Related research
Optimized LFM2.5-VL-3B: FFN Width Pruned, Distillation Aligned, INT4 Quantized
We surgically compress Liquid AI’s LFM2.5-VL-3B — an architecture-searched hybrid conv+attention vision-language model — by cutting FFN width where the search never optimized it: joint-SwiGLU Wanda pruning (10752→8192 and →7168), distillation-to-baseline LoRA recovery, and hand-rolled W4A16 GPTQ + W8 mixed-precision quantization. All three tiers are near-lossless against the base on our paired eval shards — PPL ratios at or below baseline (0.79–0.85), top-5 token agreement ≥ 0.95, quantization adding only +0.02–0.06 nats — across 5.30 GB (−15%), 4.92 GB (−21%), and 1.93 GB packed / 2.22 GB (Konic Optimized) — native INT4/INT8 compressed-tensors in vLLM (−69%). Live comparison on the smallest tier preserves scenes, facts, and tool calling — 10/10 tool rounds, including correct abstention where the base over-triggered. Recorded runs, not general benchmarks.
LFM2.5 Encoder 230M + SigLIP2: A Compact Multimodal Encoder
We augment Liquid AI’s LFM2.5 Encoder 230M — a 230M-parameter bidirectional masked-language encoder — with a SigLIP2 vision tower and 32 learned soft tokens to build an encoder-only multimodal model for retrieval and image-text matching. Clean image-only retrieval on 12,500 held-out MONET pairs reaches 0.1194 image→text R@1 with the BF16 reference; the GPTQ INT4 release retains 0.1091 while shrinking the package from 923.65 MB to 370.46 MB (−59.9%). Cyclic-negative matching AUROC is 0.9733; text-nearest hard negatives and image-conditioned masked-token prediction remain at chance, and the model does not generate.
Two Stages, Much Smaller MoE: REAP Expert Pruning Followed by AWQ INT4 Quantization
We chain two complementary compression stages on Liquid LFM2.5-8B-A1B — REAP CUDA expert pruning (32→16 experts across 22 MoE layers) then external AWQ INT4 quantization (W4A16_ASYM, group 128). The published packed artifact is 1.15B packed-weight equivalent / 2.79 GB; recorded MATH500 and BFCLv3 results use a separate 9.18 GB AWQ-scaled BF16 derivative for vLLM evaluation, not direct packed-INT4 runtime. These are recorded run summaries, not general benchmarks.