On-prem LLM terms, defined.

Terms

On-prem (on-premise)

On-premise means the model runs on infrastructure you own — your data center, private cloud, or edge hardware — under your keys, networking, and data plane. Inference happens inside your boundary, so data never leaves and cost does not scale per token.

#on-prem

Air-gapped

Air-gapped means the model, data, and serving stack run with no connection to the public internet, fully isolated inside your security perimeter. Regulated workloads that forbid external dependencies need every artifact — weights, runtime, updates — to be operable offline.

#air-gapped

Data sovereignty

Data sovereignty means an organization's data, prompts, and model outputs remain under its legal and operational jurisdiction, with no third-party API provider sharing control. Keeping models in-house keeps proprietary data inside the compliance perimeter and leaves long-term ownership of the AI system with the operator.

#data-sovereignty

Private cloud

A private cloud deployment runs the model in a dedicated VPC or tenant inside a cloud provider, rather than on the provider’s shared inference API. It keeps the data plane inside your account while letting you reuse managed networking, storage, and observability.

#private-cloud

Annual licence (no per-token cost)

An annual licence is a commercial model where a model family is licensed per host for a year and deployed on infrastructure you own. Cost does not scale with tokens served, so unit economics improve with usage instead of degrading, and spend is forecastable from the number of hosts.

#annual-licence

Quantization

Quantization stores model weights in a lower-precision format than the training precision, most commonly INT4 or INT8 instead of FP16. A well-executed INT4 build cuts weight memory roughly 4x at near-parity perplexity, which is what makes on-prem GPU footprints affordable.

#quantization

AWQ (Activation-aware Weight Quantization)

AWQ is an INT4 quantization method that scales up the weight channels aligned with large-magnitude activations before quantizing, protecting the small fraction of weights that influence outputs most. Konic publishes from-scratch AWQ implementations with reproducible perplexity results.

#awq

GPTQ

GPTQ is a post-training quantization method that quantizes weights column by column and compensates the remaining weights using second-order information from a calibration set. It is widely used to produce INT4 artifacts that load into standard GPU inference runtimes.

#gptq

Pruning

Pruning removes parameters from a trained model — whole layers, attention heads, feed-forward width, or experts — then recovers quality with continued training. Structured pruning such as FFN-width reduction shrinks the compute graph itself, so it reduces latency rather than only memory.

#pruning

REAP (Router-weighted Expert Activation Pruning)

REAP is a compression method for Mixture-of-Experts models that removes whole experts, ranked by the router-weighted magnitude of the activations they receive on a calibration set. It keeps the experts that actually fire for the target workload instead of the ones that merely hold the most weights.

#reap

Mixture-of-Experts (MoE)

A Mixture-of-Experts model replaces a dense feed-forward layer with many parallel expert networks plus a router that activates only a few per token. Total parameter count is much larger than the compute per token, which makes MoE models memory-bound and a natural target for expert pruning and quantization.

#moe

Distillation

Distillation trains a smaller or compressed model against the outputs of a larger teacher rather than against hard labels alone. Distillation alignment recovers quality after pruning, so the compressed artifact tracks the baseline output distribution instead of only matching a single loss number.

#distillation

INT4

INT4 is a 4-bit integer weight format. Weights are grouped and each group carries its own scale, which keeps precision acceptable while cutting memory to roughly a quarter of FP16. INT4 models normally need a runtime with native INT4 kernels, such as vLLM with compressed-tensors support.

#int4

vLLM

vLLM is an open-source inference server for large language models, best known for paged attention and high-throughput continuous batching. Konic publishes deployment artifacts that load directly into vLLM, including native INT4 and mixed-precision compressed-tensors builds.

#vllm

Time to first token (TTFT)

Time to first token is the latency between a request arriving and the model emitting its first output token. It is dominated by prompt prefill and is the metric users perceive as responsiveness in interactive assistant and agent workloads.

#ttft

Total cost of ownership (TCO)

Total cost of ownership for an LLM deployment includes hardware or reserved capacity, serving engineering, evaluation, monitoring, and the commercial licence — not just the per-token price of an API. On-prem deployments trade a higher fixed cost for a cost curve that flattens as usage grows.

#tco

Looking for the machine-readable version? The same definitions ship in llms-full.txt and as model pages with published benchmark tables.