Optimized LFM2.5-VL-3B: FFN Width Pruned, Distillation Aligned, INT4 Quantized

LFM2.5-VL-3B is Liquid AI’s edge-native vision-language model: 3.123B parameters, 6.25 GB in BF16, a hybrid backbone of 22 gated-convolution blocks and 8 GQA attention layers, and a SigLIP2 NaFlex vision tower. Its layer types were placed by a hardware-aware architecture search that optimized for latency — depth and block-type placement. Width was never its objective. This post records what happens when we compress the FFN width that the search left slack.

The pipeline is three cheap, well-understood stages — structured FFN channel pruning (joint-SwiGLU Wanda importance), tiny distillation LoRA recovery (KL to the cached baseline distribution), and hand-rolled mixed-precision quantization (W4A16 GPTQ + W8 RTN) — run end-to-end on a single NVIDIA L4. Three required tiers ship:

• ffn8192 — 10752→8192 FFN width, BF16, 5.30 GB (−15%)
• ffn7168 — 10752→7168, BF16, 4.92 GB (−21%)
• ffn8192-int4 — the 8192 tier quantized to 1.93 GB packed (−69%), native INT4/INT8 compressed-tensors at 2.22 GB (Konic Optimized)

The headline result: the surgical models are near-lossless against the base. On the paired eval shards, PPL ratios land at 0.79–0.85 — at or below the base’s own likelihood — top-5 token agreement stays at 0.95–0.99, and the quantization pass adds only +0.02–0.06 nats of divergence over its BF16 source. Live comparison on the smallest tier confirms the same picture end to end: identical scenes, objects, facts, and tool calls, with the Konic Optimized model sometimes more precise (see the comparison section below).

These are recorded run summaries, not general benchmarks. Heavy task evals (ScreenSpot-v2, RefCOCO) are phase gates, not yet reproduced.

The parameter budget (read directly from the checkpoint’s safetensors header) makes the target obvious:

Three structural facts shape the method:

• The FFN is a SwiGLU triple. Every layer has w1/w2/w3 [10752, 2048]; the gate couples w1/w3 rows with w2 columns, so pruning needs a joint channel criterion.
• Uniform cuts are config-compatible. block_auto_adjust_ff_dim is false in this checkpoint, so 10752 is the real dimension and a uniform width reduction is a config change (intermediate_size) plus a coupled tensor conversion — no model-code fork. 8192 and 7168 are 256-aligned.
• The convs are stateful. conv_L_cache: 3 means the recurrence carries state across positions — padding pollutes it. Every calibration and evaluation path in this project is padding-free (length-bucketed), a decision the eval harness enforces as a hard contract.

Autoresearch loops need a cheap, statistically honest answer after every surgery step: is this candidate still statistically indistinguishable from the baseline, and on which axes does it drift? The harness is a baseline reference cache + paired metrics design: the BF16 baseline runs once on fixed seeded shards (256×512-token wikitext-2 text, 128 COCO captions), per-position stats are cached, and every candidate reuses the exact same inputs against the cache — no dataset drift, no re-tokenization. ~47 s reference build, ~42 s per candidate.

The correctness contract is hard and verified:

• Bit-identity control. Identical weights must score 1.0000 on every axis — enforced by running generations first (immediately after model load) on both the reference build and candidate runs, so both sides execute in the same numeric context. Verified by three consecutive CLI self-tests: all axes exact, index 100.0.
• No PAD in conv state. Left-padded mm batches diverged 5/32 generations (measured). Text gens are uniform no-pad stacks; mm gens are grouped by exact prompt length.
• Knife-edge exclusions. Empty baseline generations sit on argmax knife-edges and carry no signal — excluded from chrF axes with the count reported (excluded_empty=6).
• GT pairing by gen_idx, never zip order (a fixed bug; the axis is now correctly sensitive: −0.11 on the projector-zero damage test).

The damage test (zeroing the multimodal projector) behaves exactly as intended: text axes are bit-identical (the projector is unused in the text path), while all mm axes deviate sharply — dnll +0.61 but kld_t64 +1.21 nats — confirming the harness detects multimodal damage where ΔNLL alone understates it.

Gates (the agreed near-lossless reference, applied by gate_check.py): PPL ratio = e^dnll ≤ 1.05 on text, ≤ 1.10 on mm. PPL ratio = 1.0 is exact parity with the base on the same inputs; anything below 1.0 means the candidate’s shard likelihood matches or exceeds the base.

The compression is near-lossless. Every tier passes the PPL-ratio gates on both shards, with ratios at or below 1.0: after distillation the pruned models’ likelihood on the same inputs matches or slightly exceeds the base’s, and the int4 tier — 69% smaller — keeps the same profile as its BF16 parent. Top-5 token agreement stays at 0.95–0.99, so the models’ highest-confidence decisions are preserved; the quantization pass adds only +0.02–0.06 nats of truncated-KL divergence over its source, i.e. within measurement noise.

The recovery arc in the Method table explains how: prune-only drifts (PPL 1.52), distillation to the baseline distribution walks likelihood back to parity, and the deeper cuts (7168) stay inside the gates with the same recipe. The comparison batteries below are the behavioral counterpart — the same facts, scenes, and tool calls on real prompts.

The packed artifact is converted to a compressed-tensors checkpoint that runs natively in vLLM 0.26.0 on the same L4 — no dequant, real INT4/INT8 weights (code/serve/to_ct_format.py, values verified bit-exact against the harness-evaluated checkpoint).

• Dequant-eval caveat. The harness evaluates exact dequant numerics, not quantized kernels; real-kernel evidence is the vLLM comparison batteries.
• Heavy task benchmarks pending. ScreenSpot-v2 (80.7), RefCOCO (87.9), MME/MMStar are vendor claims being reproduced at phase gates via lm-evaluation-harness — the quick suite is the frequent-loop instrument, not the final word.
• Vision drop deferred. Dropping the vision tower’s top-3 layers without projector retrain shifts the mm distribution measurably; shipping it requires the retrain (phase 3).
• Single recorded runs. Point estimates (plus bootstrap CIs on the harness axes) carry unknown single-run noise; tool batteries are 10 curated rounds.

All three tiers are public under konic-labs in the “LFM2.5 3B Vision - Optimized” collection (LFM Open License v1.0 retained). The pipeline is committed and runnable on one L4:

# 0. Build the baseline reference cache once (paired stats, ~47 s)
PYTHONPATH=code /data/venv/bin/python3 -m eval_harness.cli build-reference \
  --suite quick --baseline <bf16_dir> --tag lfm25-vl-3b-bf16

# 1. Prune: joint-SwiGLU Wanda, uniform tier (scores reusable across tiers)
python3 -m pruning.wanda_prune --baseline <bf16_dir> --out <pruned_dir> \
  --target 8192 --text-samples 256 --mm-samples 32

# 2. Recover: KL distillation to the cached baseline distribution (top-64)
python3 -m pruning.cache_teacher --baseline <bf16_dir> \
  --manifest text_train_disjoint_2048.jsonl --out teacher_disjoint2048_topk.npz
python3 -m pruning.lora_recover --model <pruned_dir> --out <rec_dir> \
  --distill teacher_disjoint2048_topk.npz \
  --train-manifest text_train_disjoint_2048.jsonl

# 3. Quantize: W4A16 GPTQ + W8 RTN + int8 embeddings
python3 -m pruning.quantize --model <rec_dir> --out <quant_dir> --calib-samples 128

# 4. Eval any candidate against the cache
PYTHONPATH=code /data/venv/bin/python3 -m eval_harness.cli run \
  --suite quick --candidate <candidate_dir> --candidate-tag <name>

Tested environment: NVIDIA L4 (23 GB, sm_89), 4 vCPU / 15 GiB RAM, PyTorch 2.11.0+cu128, Transformers 5.14.1, vLLM 0.26.0 (flashinfer JIT with FLASHINFER_DISABLE_VERSION_CHECK=1), Datasets 5.0, Python 3.10.12. Training manifests are pinned JSONL and verified disjoint from the eval shard; teacher caches are committed alongside.

The width-slack hypothesis is supported, and the surgical compression is near-lossless in practice: 24% FFN width removal with small distillation LoRA lands at 5.30 GB, the 33% cut at 4.92 GB, and hand-rolled W4A16 GPTQ + W8 RTN carries the 8192 tier to 1.93 GB packed (−69%) — native INT4/INT8 compressed-tensors at 2.22 GB (Konic Optimized). All three tiers sit at or below baseline shard likelihood (PPL ratios 0.79–0.85, gates pass), keep top-5 token agreement at 0.95–0.99, and the quantization pass adds only +0.02–0.06 nats over its BF16 parent. Live comparison on the smallest tier reproduces the same scenes, facts, and tool calls as the base — 10/10 tool rounds, including correct abstention on no-tool prompts where the base over-triggered.

Remaining work is the phase-gate heavy benchmarks (ScreenSpot-v2, RefCOCO, MME/MMStar), the deferred vision-drop retrain, and the FP8 vs W4A16 head-to-head. Teams evaluating these artifacts should measure on their own distributions rather than extrapolate from these recorded runs.

• Collection: konic-labs — LFM2.5 3B Vision - Optimized
• Releases: LFM2.5-VL-3B-ffn8192 (5.30 GB) · ffn7168 (4.92 GB) · ffn8192-int4 (2.57 GB, compressed-tensors, vLLM-ready)
• Base model: LiquidAI/LFM2.5-VL-3B — 3.123B params, 22 conv + 8 attention layers, SigLIP2 so400m NaFlex vision tower; LFM Open License v1.0.
• Evaluation harness: code/eval_harness/ in the lab repo — baseline reference cache + paired metrics, bit-identity control, block bootstrap CIs.
• Compression pipeline: code/pruning/ —wanda_prune.py, lora_recover.py, quantize.py + quant_loader.py; compressed-tensors conversion code/serve/to_ct_format.py; comparison batteries code/serve/compare_vllm.py and compare_tools2.py.
• Prior art: Wanda (Sun et al., 2023); GPTQ (Frantar et al., 2022); LoRA (Hu et al., 2021); POP prefill-only pruning (arXiv 2602.03295); SigLIP2 NaFlex (google/siglip2-so400m-patch16-naflex).