On-Prem LLM vs API: When to Self-Host Your Language Models
TL;DR
API-based LLMs are fast to adopt but scale poorly on cost, latency, and data control. On-prem LLMs trade setup effort for predictable economics, lower per-token cost at volume, and data that never leaves your boundary. This guide breaks down the decision across cost, latency, privacy, compliance, and operational effort — and when each path is the right call.
The two ways to run a language model
Every team that puts an LLM into production eventually faces the same choice: call a hosted API, or run the model on infrastructure you control. The API path is the fastest way to a working demo. The on-prem path is the way most enterprises end up running models at meaningful production volume — because the economics, latency, and data-control profile of an API change as soon as usage stops being trivial.
This guide compares the two honestly, across the dimensions that actually decide the question: cost, latency, privacy and compliance, operational effort, and the ability to customize the model itself.
Cost: the API bill compounds with usage
An API is priced per token. That is simple to understand and cheap at low volume — which is exactly why it feels right at the start. The problem is that production AI usage does not stay low. Agentic workflows, retrieval pipelines, and multi-step reasoning all multiply the number of tokens per task. A single agent run can consume tens of thousands of tokens across tool calls and retries.
On-prem deployment replaces the per-token meter with a fixed cost: the hardware, the model licence, and the engineering to run it. Once you are past the break-even point, the marginal cost of an extra request approaches zero. For workloads that run continuously — the kind that actually matter in production — that difference is the whole story.
Latency and throughput: your data plane, your rules
Every API call crosses a network boundary and queues behind shared capacity. That adds latency and, more importantly, makes latency variable — a problem for agentic systems that make many sequential calls and for real-time features where a slow tail degrades the product.
Running the model on your own hardware puts inference on your data plane. You control the serving stack, the batch size, the quantization, and the concurrency. For latency-sensitive or high-throughput workloads, on-prem is not a preference — it is the only way to hit the target.
Privacy, compliance, and data sovereignty
This is the dimension where the API model breaks down for regulated industries. Sending proprietary data to a hosted API means that data leaves your environment — even if the provider promises not to train on it, the data plane is not yours. For banking, insurance, healthcare, and manufacturing, that is often a hard stop.
On-prem deployment keeps the model and the data inside your boundary. That is what makes air-gapped and sovereign deployments possible: the model runs on machines you own, behind your network, with your keys. If your compliance posture requires data to stay in a jurisdiction or on a specific network, on-prem is not optional.
Customization: the model is the product
An API gives you a fixed model. You can prompt it, but you cannot change what it is. On-prem gives you the weights — which means you can fine-tune, prune, quantize, and specialize the model for your workload. That is the difference between adapting to a general model and building a model that fits your task.
For teams running high-volume, repetitive production behavior, a smaller, task-specific model that is fine-tuned and compressed for their hardware can match or beat a much larger general model — at a fraction of the footprint and cost.
Operational effort: the honest trade-off
The API path wins on setup. There is no hardware, no serving stack, no model to maintain. That is real value, and it is the right choice for experiments, prototypes, and low-volume internal tools.
The on-prem path requires you to own the serving layer: the runtime (vLLM, MLX, llama.cpp), the quantization, the monitoring, and the upgrades. That is why most enterprises do not self-host a raw checkpoint — they work with a vendor that ships a production-shaped artifact and the engineering to run it.
When to choose which
- Choose an API when you are prototyping, when volume is low and unpredictable, or when the data is not sensitive and latency is not critical.
- Choose on-prem when usage is steady and high, when data must stay in your boundary, when latency is a product requirement, or when you need a model specialized to your task.
Most enterprises end up with a hybrid: APIs for exploration, on-prem for the production workloads that matter. The question is not which one to use — it is which workloads belong on which path.
More research
Optimized LFM2.5-VL-3B: FFN Width Pruned, Distillation Aligned, INT4 Quantized
We surgically compress Liquid AI’s LFM2.5-VL-3B — an architecture-searched hybrid conv+attention vision-language model — by cutting FFN width where the search never optimized it: joint-SwiGLU Wanda pruning (10752→8192 and →7168), distillation-to-baseline LoRA recovery, and hand-rolled W4A16 GPTQ + W8 mixed-precision quantization. All three tiers are near-lossless against the base on our paired eval shards — PPL ratios at or below baseline (0.79–0.85), top-5 token agreement ≥ 0.95, quantization adding only +0.02–0.06 nats — across 5.30 GB (−15%), 4.92 GB (−21%), and 1.93 GB packed / 2.22 GB (Konic Optimized) — native INT4/INT8 compressed-tensors in vLLM (−69%). Live comparison on the smallest tier preserves scenes, facts, and tool calling — 10/10 tool rounds, including correct abstention where the base over-triggered. Recorded runs, not general benchmarks.
LFM2.5 Encoder 230M + SigLIP2: A Compact Multimodal Encoder
We augment Liquid AI’s LFM2.5 Encoder 230M — a 230M-parameter bidirectional masked-language encoder — with a SigLIP2 vision tower and 32 learned soft tokens to build an encoder-only multimodal model for retrieval and image-text matching. Clean image-only retrieval on 12,500 held-out MONET pairs reaches 0.1194 image→text R@1 with the BF16 reference; the GPTQ INT4 release retains 0.1091 while shrinking the package from 923.65 MB to 370.46 MB (−59.9%). Cyclic-negative matching AUROC is 0.9733; text-nearest hard negatives and image-conditioned masked-token prediction remain at chance, and the model does not generate.
Two Stages, Much Smaller MoE: REAP Expert Pruning Followed by AWQ INT4 Quantization
We chain two complementary compression stages on Liquid LFM2.5-8B-A1B — REAP CUDA expert pruning (32→16 experts across 22 MoE layers) then external AWQ INT4 quantization (W4A16_ASYM, group 128). The published packed artifact is 1.15B packed-weight equivalent / 2.79 GB; recorded MATH500 and BFCLv3 results use a separate 9.18 GB AWQ-scaled BF16 derivative for vLLM evaluation, not direct packed-INT4 runtime. These are recorded run summaries, not general benchmarks.
From-Scratch AWQ INT4 Quantization on Qwen3-8B
We validate a from-scratch, pure-PyTorch AWQ implementation on Qwen3-8B: group-wise INT4 with per-channel AWQ scaling produces a 4.0× smaller model (13.9 GB → 3.5 GB linear weights) at 1.034× FP16 perplexity on WikiText-2 (10.08 vs 9.75), loaded and run in a real INT4 GEMM runtime. The decisive factor is norm-folding the AWQ scale — 20× more accurate per weight than weight-dequantization.
REAP Expert Pruning for On-Device MoE Models
We present REAP-MLX, an Apple Silicon implementation of Router-weighted Expert Activation Pruning (REAP) for Mixture-of-Experts language models. We evaluate quality retention across compression ratios on the LFM2.5-8B architecture: REAP preserves 96.8% of code generation performance at 25% compression and 91.4% at 50% compression, with less than 0.4 percentage point variance across independent calibration draws.