How to Deploy an LLM On-Premise: A Practical Guide

What is on-prem LLM deployment? It is running a language model on infrastructure you own, with your keys and your data plane. All inference, data, and traces stay inside your boundary with no call out to a third-party API.

Self-hosting a language model is not one task — it is a chain of decisions, each with a real trade-off. Get the model right and the hardware wrong and you pay for idle GPUs. Get the runtime right and the quantization wrong and you leave latency on the table. This guide walks the full sequence so you can make each call deliberately instead of by default.

The reason to think in steps is that self-hosting shifts ownership to you. You choose the model weights, you provision the accelerators, you operate the serving stack, and you own availability and upgrades. That is the point — control over cost, latency, and where data flows — but it also means each decision compounds. A model that is too large forces more VRAM, which forces a larger runtime topology, which changes how you quantize. Making each choice in order keeps you from rediscovering the constraints after you have already committed to hardware.

This guide stays practical. It does not prescribe a single stack for every team; it gives you the checks at each gate so you can size and ship a deploy that fits your workload, not someone else's benchmark.

The single most common mistake is choosing the largest model you can find. A frontier model sized for every possible task is overkill for a workload that does one thing well. Start from the task: what does the model actually need to do, and what accuracy bar does the workload set?

A smaller model that is fine-tuned and compressed for your specific workload will often match a much larger general model on the task that matters — at a fraction of the memory, latency, and cost. Size the model to the job.

In practice this means scoping the task to measurable outputs: the document types you handle, the tools the model must call, the latency you must hit at P95, and the error modes you cannot tolerate. A 3B vision-language model that is pruned and distilled for your document workflow can beat a 70B general model on that workflow while fitting on a single workstation GPU. The earlier you define the accuracy bar on real prompts, the earlier you can rule out oversized models and avoid paying for capacity you will never use.

When you do compare, compare on your eval set, not a leaderboard. Run the candidate models against your own prompts with the decoding settings you will ship, measure task success and not just perplexity, and keep the smallest model that clears the bar. That discipline also gives you a valid baseline to measure compression against later — so you know whether a quantized variant actually preserved what you needed.

The model's parameter count and precision determine your memory floor. A rough rule: a model with N billion parameters in FP16 needs roughly 2N GB of VRAM just for the weights, plus headroom for the KV cache and activations. Quantization changes that math dramatically — an INT4 model needs roughly a quarter of the memory of its FP16 equivalent.

Your latency target then decides how many GPUs and how much concurrency you need. Batch size, sequence length, and throughput all feed the same equation. Size for the workload you actually run, not the theoretical maximum.

Quantization is the difference between needing an 80 GB reservation and fitting on an edge workstation. Our Konic LFM2.5-VL-3B study (2026-08-13, INT4 6.25GB→1.93GB −69%) shows how large that shift can be: INT4 compression took the model from 6.25 GB to 1.93 GB — a 69% reduction — with near-lossless quality on the paired eval shards (LFM2.5-VL-3B optimized). That is not a toy delta; it is the difference between one model per GPU and multiple replicas per card.

To size correctly, start with weights, then add KV cache for your max context and batch, then add runtime overhead. A 7–8B model at FP16 wants roughly 16 GB for weights alone; the same model in INT4 wants roughly 4 GB. The headroom you keep for cache determines how many concurrent requests you can serve before throughput collapses — which is why the table in the appendix below maps concrete size points to FP16 and INT4 footprints by runtime.

The runtime is the layer that turns weights into a usable service. The main options:

• vLLM — high-throughput serving with continuous batching and PagedAttention. The default for GPU production serving.
• MLX — Apple Silicon optimized. The right choice for on-device and edge deployments on Mac hardware.
• llama.cpp — lightweight, CPU-friendly, and portable. Good for constrained environments and quick tests.

Your choice depends on your hardware and your throughput needs. For GPU-heavy production, vLLM is the standard starting point.

Each runtime makes a different trade. vLLM maximizes GPU throughput and works natively with AWQ and compressed-tensors layouts, which is why it is the default for on-prem production behind an API. MLX is the right path when you deploy on Apple Silicon and want the model close to the user with minimal host overhead. llama.cpp is the portability layer — easy to run on a laptop or CI runner, trivial to bring offline, but not built for high-concurrency serving. Pick one primary runtime for production and keep a lightweight alternative for local testing; supporting two production runtimes is rarely worth the surface.

When you choose, also check the quantization formats the runtime loads without conversion. If your compression target is INT4 AWQ, confirm the runtime supports AWQ GEMM or compressed-tensors W4A16 and that your export packs the scales the runtime expects. The wrong format can turn a near-lossless model into a slow or lossy deploy.

Quantization reduces the precision of the weights to fit more model into less memory and run faster. The two approaches that matter in practice:

• Post-training quantization (PTQ) — like AWQ or GPTQ — compresses an existing model without retraining. Fast to apply, and with good calibration it preserves most of the quality.
• Pruning removes parameters the model does not use, shrinking the model itself before quantization.

The right recipe depends on your model and hardware. The key is to measure quality after compression, not assume it — a well-calibrated INT4 model can be near-lossless on your task.

For a concrete calibration on what PTQ can deliver, see our AWQ INT4 on Qwen3-8B study: a from-scratch AWQ pipeline produced a 4.0x smaller model at 1.034x perplexity versus FP16, with greedy outputs indistinguishable in substance in a real INT4 GEMM runtime. That result is from a fully inspectable pure-PyTorch implementation, not a black-box toolkit — every scale search, packing, and runtime handoff is auditable.

In your own deploy, pair the method to the model family. AWQ-style activation-aware scaling is strongest for dense and MoE LLMs where a few salient channels dominate quantization error; GPTQ-style second-order correction is strongest when you can afford a heavier calibration set. For vision-language or MoE workloads where expert redundancy is high, combine light structural pruning first, then quantize — which is exactly the ffn pruning plus INT4 route that delivered the 69% footprint cut cited above.

An on-prem model is software you now operate. That means you need the same discipline as any production service: request logging, latency and throughput metrics, error tracking, and — most importantly — an evaluation loop that tells you whether the model is still meeting the accuracy bar as you upgrade it.

The teams that succeed with on-prem treat the model as a versioned product: they A/B test a new release against the incumbent on a real workload and decide on measured results, not claims.

Build the loop before you scale traffic. Log prompts, tool calls, and completions with request IDs, export latency histograms by sequence length, and run a nightly eval on a frozen slice of real traffic where you have graded expectations. When you ship a new compressed variant, promote it only if it matches the incumbent on that slice — otherwise you are trading invisible quality loss for visible latency wins. Monitoring also surfaces the signals that tell you when to re-quantize or grow the replica count rather than guess.

Evaluation should run where the model runs. Keep the eval harness inside your boundary so you are not sending production prompts back to a vendor. That keeps the data plane consistent from inference to measurement and makes offline version comparisons meaningful.

The steps above are exactly what a model vendor does for you. A production-shaped artifact — already pruned, quantized, and validated for a serving runtime — removes most of the engineering risk. If you would rather evaluate a model on your hardware than build the pipeline yourself, that is the path most enterprises take.

Konic ships that shortcut as a production-shaped artifact: models are delivered pruned, quantized, and validated for a concrete runtime like vLLM or MLX, not as a raw checkpoint you must operationalize. You load the artifact, point the runtime at it, and run the eval that matters to you — without rebuilding calibration, packing, or scale-folding from scratch.

This is the fastest way to answer the buy-versus-build question honestly. Evaluate the artifact on your hardware in your network, measure it against your accuracy bar, and decide with numbers. If it clears the bar, you have skipped months of compression engineering and can focus on wiring evaluation and operations — which are where on-prem success is actually decided.

Use this matrix as a starting point and then size for your workload's batch and context. FP16 is the raw weight footprint; INT4 is the weight footprint after AWQ or equivalent W4A16 compression. The runtime column names the minimal production runtime that loads the format without conversion.

The 3.1B row reflects the measured LFM2.5-VL-3B artifact (6.25 GB → 1.93 GB) cited above; the other rows are the 2N rule rounded for common model families. Your KV cache for long contexts can add several gigabytes at higher batch — size it explicitly before you lock GPU count.

Once you have an AWQ INT4 export, serving it is one command. This example loads a production-shaped INT4 artifact with vLLM's AWQ quantization path, pins FP16 activations, and leaves room for KV cache at high concurrency:

Point --quantization awq at an artifact that was packed for the AWQ GEMM layout and whose preceding-norm scales were folded — exactly the export validated in the AWQ INT4 on Qwen3-8B study. From there scale replicas, not quantization settings.

How do I size VRAM for a self-hosted model?

Start with the 2N rule: a model with N billion parameters needs roughly 2N GB of VRAM in FP16 for weights alone, then add headroom for KV cache, activations, and the runtime. For a 7B model that is about 14 GB for weights; for 13B about 26 GB. INT4 changes the equation to roughly 0.5N GB — the 3.1B LFM2.5-VL-3B INT4 artifact at 1.93 GB is a measured example. Size cache for your max sequence length and concurrency, not the average; a batch of long contexts can add several gigabytes beyond the weight footprint, which is why the sizing matrix above lists both FP16 and INT4.

How much accuracy do I lose with INT4?

With good calibration, very little. The AWQ INT4 on Qwen3-8B study is 4.0x smaller at 1.034x perplexity with greedy outputs that are indistinguishable in substance from FP16 in a real INT4 GEMM runtime. Separately, the LFM2.5-VL-3B optimized study kept near-lossless quality on paired eval shards at a 69% footprint reduction (6.25 GB → 1.93 GB). The key is to measure on your task after compression — PPL ratios, top-5 token agreement, and task success on your eval set — rather than assume loss from a generic benchmark.

Which runtime should I choose?

Pick the runtime your hardware already favors. For GPU production behind an API, choose vLLM — it is the default for high-throughput serving with continuous batching and native AWQ/compressed-tensors support. For Apple Silicon and on-device edge, choose MLX. For lightweight or CPU-only portability, choose llama.cpp. Validate that the runtime loads your quantized format natively, then commit to one production runtime and keep the other as a local testing layer. Changing runtimes after you have tuned quantization forces a repack you do not want.