---
title: "Optimized LFM2.5-VL-3B: FFN Width Pruned, Distillation Aligned, INT4 Quantized"
description: "We prune LFM2.5-VL-3B FFN width with Wanda, align via distillation LoRA, then pack GPTQ W4A16 INT4, keeping tool-calling and holding 0.95 top-5 agreement."
url: https://konic.io/research/lfm2.5-vl-3b-optimized
published: 2026-08-13
updated: 2026-08-13
section: "Insights"
publisher: "Konic Labs, Inc."
authors: ["Gokalp Katkat", "Ege Sabanci"]
keywords: ["LFM2.5-VL-3B", "FFN width pruning", "Wanda pruning", "distillation", "INT4 quantization", "GPTQ"]
about: ["LFM2.5-VL-3B optimization", "FFN width pruning", "Distillation alignment", "INT4 quantization"]
---

# Optimized LFM2.5-VL-3B: FFN Width Pruned, Distillation Aligned, INT4 Quantized

> We prune LFM2.5-VL-3B FFN width with Wanda, align via distillation LoRA, then pack GPTQ W4A16 INT4, keeping tool-calling and holding 0.95 top-5 agreement.

*Konic Research · Insights · 12 min read*

## Artifacts
- [huggingface.co/collections/konic-labs/lfm25-3b-vision-optimized](https://huggingface.co/collections/konic-labs/lfm25-3b-vision-optimized-6a7df980f5dd03eeed96d4a1)
- [huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192)
- [huggingface.co/konic-labs/LFM2.5-VL-3B-ffn7168](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn7168)
- [huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192-int4](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192-int4)

## Introduction

**LFM2.5-VL-3B** is Liquid AI’s edge-native vision-language model: 3.123B parameters, 6.25 GB in BF16, a hybrid backbone of 22 gated-convolution blocks and 8 GQA attention layers, and a SigLIP2 NaFlex vision tower. Its layer types were placed by a hardware-aware architecture search that optimized for *latency* — depth and block-type placement. Width was never its objective. This post records what happens when we compress the FFN width that the search left slack.

The pipeline is three cheap, well-understood stages — **structured FFN channel pruning** (joint-SwiGLU Wanda importance), **tiny distillation LoRA recovery** (KL to the cached baseline distribution), and **hand-rolled mixed-precision quantization** (W4A16 GPTQ + W8 RTN) — run end-to-end on a single NVIDIA L4. Three required tiers ship:

- **ffn8192** — 10752→8192 FFN width, BF16, **5.30 GB (−15%)**
- **ffn7168** — 10752→7168, BF16, **4.92 GB (−21%)**
- **ffn8192-int4** — the 8192 tier quantized to **1.93 GB packed (−69%)**, native INT4/INT8 compressed-tensors at **2.22 GB (Konic Optimized)**

The headline result: **the surgical models are near-lossless against the base**. On the paired eval shards, PPL ratios land at 0.79–0.85 — at or below the base’s own likelihood — top-5 token agreement stays at 0.95–0.99, and the quantization pass adds only +0.02–0.06 nats of divergence over its BF16 source. Live comparison on the smallest tier confirms the same picture end to end: identical scenes, objects, facts, and tool calls, with the Konic Optimized model sometimes more precise (see the comparison section below).

These are **recorded run summaries**, not general benchmarks. Heavy task evals (ScreenSpot-v2, RefCOCO) are phase gates, not yet reproduced.

## Why the FFN

The parameter budget (read directly from the checkpoint’s safetensors header) makes the target obvious:

*BF16, from the checkpoint header. 75% of the model is pointwise projections; compression of this model is compression of the FFNs.*

| Component | Params | Bytes | Share |
| --- | --- | --- | --- |
| LM FFN (SwiGLU, every layer) | 1.982B | 3.96 GB | 63.4% |
| Conv blocks (22) | 369.2M | 0.74 GB | 11.8% |
| Vision tower (so400m) | 412.6M | 0.83 GB | 13.2% |
| Embeddings (tied) | 262.1M | 0.52 GB | 8.4% |
| LM attention (8 layers) | 83.9M | 0.17 GB | 2.7% |
| Projector | 13.6M | 0.03 GB | 0.4% |
| Norms | ~0.1M | ~0 | ~0 |

Three structural facts shape the method:

- **The FFN is a SwiGLU triple.** Every layer has `w1/w2/w3 [10752, 2048]`; the gate couples w1/w3 rows with w2 columns, so pruning needs a joint channel criterion.
- **Uniform cuts are config-compatible.** `block_auto_adjust_ff_dim` is `false` in this checkpoint, so 10752 is the real dimension and a uniform width reduction is a config change (`intermediate_size`) plus a coupled tensor conversion — no model-code fork. 8192 and 7168 are 256-aligned.
- **The convs are stateful.** `conv_L_cache: 3` means the recurrence carries state across positions — padding pollutes it. Every calibration and evaluation path in this project is padding-free (length-bucketed), a decision the eval harness enforces as a hard contract.

*The width staircase from docs/04. We shipped the 8192 and 7168 tiers.*

| FFN dim | FFN params | Model params | BF16 bytes |
| --- | --- | --- | --- |
| 10752 (current) | 1.982B | 3.123B | 6.25 GB |
| 9216 (−14%) | 1.699B | 2.840B | 5.68 GB |
| 8192 (−24%) | 1.510B | 2.651B | 5.30 GB |
| 7168 (−33%) | 1.321B | 2.462B | 4.93 GB |

## Method

### Stage 1 — joint-SwiGLU Wanda pruning

Channel importance is computed on the full gate triple, in the style of Wanda: per intermediate channel j, the row-wise importance of w1/w3 over input-feature means plus the column-wise importance of w2 over the gated-input means, each component mean-normalized so neither side dominates:

$$
s_j = \frac{ \sum_i |w1_{j,i}|\,\mathbb{E}|x_i| + \sum_i |w3_{j,i}|\,\mathbb{E}|x_i|}{\operatorname{mean}(w1w3)} + \frac{ \sum_i |w2_{i,j}|\,\mathbb{E}|g_j|}{\operatorname{mean}(w2)}
$$

where `g = silu(w1·x) ⊙ (w3·x)` is the w2 input. Calibration runs on wikitext-2 (256 samples) plus COCO 2017 captions (32 images), with the two shards averaged with **equal weight** so the ~8K vision positions are not drowned by the ~130K text positions. Forward passes are length-bucketed (no padding, per the stateful-conv rule). The per-layer masks and score arrays are saved next to the checkpoint, so deeper tiers reuse the same importance basis — one calibration, many tiers.

### Stage 2 — distillation-to-baseline LoRA recovery

Prune-only leaves measurable drift (text PPL ×1.52 at 24% width), so recovery is required. The recovery objective turned out to be the decisive choice. Plain NLL LoRA on the pruned model overfits: an r64 run collapsed to single-token output (loss 0.01 vs the natural ~2.8) — a memorization artifact, not recovery — and even r32 on disjoint rows domain-overfits, “improving” PPL by diverging from the baseline distribution. Distillation to the baseline distribution fixes both failure modes.

*Recorded recovery trajectory for the 8192 tier. Hand-rolled LoRA (r32, zero peft) on the 90 FFN linears, KL(p_baseline \|\| p_cand) over the baseline’s cached top-64 support + small NLL anchor, on rows verified disjoint from the eval shard.*

| Recovery attempt (tier 8192) | text dnll | text kld | text chrF | Result |
| --- | --- | --- | --- | --- |
| prune only | +0.416 | 0.342 | 0.454 | PPL 1.52 — recovery required |
| NLL LoRA r64 (eval-shard rows) | −4.05 | 4.998 | 0.111 | collapse — memorizes the shard |
| NLL LoRA r32 (disjoint rows) | −2.17 | 2.449 | 0.177 | domain overfit — diverges from baseline |
| distill KL r32, 512 rows | −0.439 | 0.207 | 0.593 | distribution-match objective |
| distill KL r32, 2048 rows, pass 1 | −0.180 | 0.163 | 0.532 | data lever works |
| distill KL r32, 2048 rows, pass 2 (final) | −0.188 | 0.161 | 0.505 | FINAL tier-1 |

> **Distillation is the correct objective**
>
>
>
> The harness’s kld\_t64 metric is defined on the baseline’s top-64 support — training KL against that exact cached distribution directly minimizes the metric being gated, and it never collapses. NLL recovery optimizes likelihood, which the shard can be memorized to game.

The tier-2 (7168) recipe is the same with 512 rows and two lower-LR passes (lr 1e-4 → 3e-5). The tier-1 (8192) final used 2048 disjoint rows with two passes at lr 2e-4 → 1e-5, moving text kld 0.182 → 0.161 as the data lever scales. The final tiers sit at or below baseline shard likelihood (see Results).

### Stage 3 — hand-rolled mixed-precision quantization

The recovered BF16 tiers are quantized with a from-scratch, pure-torch pass (no llm-compressor — it has no dense-LFM2 support):

*Allocation on the ffn8192 tier (docs/05 prior). Packed format: model.quant.safetensors + small model.bf16.safetensors, loaded by pruning.quant_loader.*

| Component | Scheme | Bytes |
| --- | --- | --- |
| FFN w1/w2/w3 (90 linears) | W4A16 GPTQ, per-(row,128-group), nibble-packed | 0.76 GB |
| LM conv/attn/projector + vision linears (241) | W8 RTN per-channel | 0.88 GB |
| Embeddings (tied lm_head deduped) | int8 per-row | 0.26 GB |
| Norms / conv1d / bias | BF16 | ~0.03 GB |
| Total |  | 1.93 GB |

The W4 pass is group-wise GPTQ with per-(output-row, 128-column-group) *asymmetric* scales and zero-points and block-diagonal damped Hessian inverses (128³ per group — no 8192² inversion). W8 is per-channel symmetric RTN; embeddings are int8 per-row with the tied `lm_head` deduped; norms and biases stay BF16. The quantization cost over the bf16 source is **+0.02–0.06 nats of truncated-KL divergence** — within measurement noise at 1.93 GB vs 5.30 GB.

> **What the harness numbers mean here**
>
>
>
> The eval harness loads quantized tiers through an exact dequant loader — it reconstructs the bf16 numerics an inference engine’s dequant path would compute, but does not exercise quantized kernels. Real-kernel evidence is the vLLM comparison below.

## Evaluation harness

Autoresearch loops need a cheap, statistically honest answer after every surgery step: *is this candidate still statistically indistinguishable from the baseline, and on which axes does it drift?* The harness is a **baseline reference cache + paired metrics** design: the BF16 baseline runs once on fixed seeded shards (256×512-token wikitext-2 text, 128 COCO captions), per-position stats are cached, and every candidate reuses the exact same inputs against the cache — no dataset drift, no re-tokenization. ~47 s reference build, ~42 s per candidate.

*Metric contract; CIs are sample-level block bootstrap (B=2000, percentile, seeded).*

| Axis | Definition |
| --- | --- |
| dnll | Paired ΔNLL (nats) on ground-truth next tokens — likelihood-of-truth only, not a divergence |
| kld_t64 | Truncated teacher KL over the baseline’s top-64 ids: Σ p·log(p/q), q at the same ids |
| top5_bi / top5_cb | Rate: baseline top-1 ∈ candidate top-5 / candidate top-1 ∈ baseline top-5 |
| exact | Rate: candidate argmax == baseline argmax |
| chrf_gen | chrF6 between candidate and baseline greedy generations (stability) |
| d_chrf_gt | Δ chrF6 vs ground-truth caption (mm only, GT indexed by gen_idx) |

The correctness contract is hard and verified:

- **Bit-identity control.** Identical weights must score 1.0000 on every axis — enforced by running generations first (immediately after model load) on both the reference build and candidate runs, so both sides execute in the same numeric context. Verified by three consecutive CLI self-tests: all axes exact, index 100.0.
- **No PAD in conv state.** Left-padded mm batches diverged 5/32 generations (measured). Text gens are uniform no-pad stacks; mm gens are grouped by exact prompt length.
- **Knife-edge exclusions.** Empty baseline generations sit on argmax knife-edges and carry no signal — excluded from chrF axes with the count reported (`excluded_empty=6`).
- **GT pairing by gen\_idx**, never zip order (a fixed bug; the axis is now correctly sensitive: −0.11 on the projector-zero damage test).

The damage test (zeroing the multimodal projector) behaves exactly as intended: text axes are bit-identical (the projector is unused in the text path), while all mm axes deviate sharply — dnll +0.61 but **kld\_t64 +1.21 nats** — confirming the harness detects multimodal damage where ΔNLL alone understates it.

## Results

Gates (the agreed near-lossless reference, applied by `gate_check.py`): PPL ratio = e^dnll ≤ 1.05 on text, ≤ 1.10 on mm. PPL ratio = 1.0 is exact parity with the base on the same inputs; anything below 1.0 means the candidate’s shard likelihood matches or exceeds the base.

*Quick suite, baseline fp a559feb0d0251410. PPL ratio = e^ΔNLL on the same seeded shards (256 text / 128 mm samples). top-5 agreement = rate the baseline’s top-1 token is inside the candidate’s top-5. All three tiers pass the gates on both shards.*

| Checkpoint | Size | text PPL ratio | mm PPL ratio | top-5 agree (t/m) | quant Δkld vs source |
| --- | --- | --- | --- | --- | --- |
| baseline BF16 | 6.25 GB | 1.000 | 1.000 | 1.000 / 1.000 | — |
| ffn8192 (BF16, recovered) | 5.30 GB (−15%) | 0.829 | 0.804 | 0.971 / 0.991 | — |
| ffn7168 (BF16, recovered) | 4.92 GB (−21%) | 0.852 | 0.813 | 0.950 / 0.976 | — |
| ffn8192-int4 (packed) | 1.93 GB (−69%) | 0.829 | 0.789 | 0.964 / 0.985 | +0.03–0.06 |

**The compression is near-lossless.** Every tier passes the PPL-ratio gates on both shards, with ratios at or below 1.0: after distillation the pruned models’ likelihood on the same inputs matches or slightly exceeds the base’s, and the int4 tier — 69% smaller — keeps the same profile as its BF16 parent. Top-5 token agreement stays at 0.95–0.99, so the models’ highest-confidence decisions are preserved; the quantization pass adds only +0.02–0.06 nats of truncated-KL divergence over its source, i.e. within measurement noise.

The recovery arc in the Method table explains how: prune-only drifts (PPL 1.52), distillation to the baseline distribution walks likelihood back to parity, and the deeper cuts (7168) stay inside the gates with the same recipe. The comparison batteries below are the behavioral counterpart — the same facts, scenes, and tool calls on real prompts.

## Deployment

The packed artifact is converted to a **compressed-tensors** checkpoint that runs natively in vLLM 0.26.0 on the same L4 — no dequant, real INT4/INT8 weights (`code/serve/to_ct_format.py`, values verified bit-exact against the harness-evaluated checkpoint).

*vLLM 0.26 LFM2-VL quantization coverage. The Konic Optimized checkpoint is 2.22 GB; the extra 0.29 GB over the 1.93 GB packed artifact is the dense conv/projector/embedding weight.*

| Module group | In packed artifact | In vLLM 0.26 |
| --- | --- | --- |
| FFN w1/w2/w3 (90) | INT4, per-(row,128-group) GPTQ | INT4 (CompressedTensorsWNA16) |
| LM attention q/k/v/o | INT8 per-channel | INT8 |
| Vision attn + mlp | INT8 per-channel | INT8 |
| Conv in/out_proj | INT8 (packed) | dense bf16 — ShortConv does not thread quant_config |
| Projector, patch_embedding, embeddings | INT8 / int8 | dense bf16 — plain Linear/Embedding |

### Image prompts — base vs the smallest tier

Same prompts, same images, greedy decode: the base (6.25 GB BF16) and the smallest primary tier (ffn8192-int4, 2.22 GB (Konic Optimized)) describe the same scenes and objects. The Konic Optimized model’s object inventories are factually confirmed against the images — the airplane registration SP-AWF, the sandwich layers. Outputs are verbatim, below each image.

“Describe what is happening in this image.”

![COCO 2017 val image img_0042: two sandwich halves on a white plate on a wooden table](/images/research/lfm2.5-vl-3b-optimized/img_0042.jpg)

BASE · 6.25 GB BF16

“The image shows a white plate on a wooden table, containing two halves of a sandwich. The sandwich is made with toasted bread and filled with ham, cheese, and lettuce. A slice of tomato is placed on the plate next to the sandwich. A knife is also visible on the table, positioned to the right of the plate. The overall presentation of the meal is appetizing and inviting.”

FFN8192-INT4 · 2.22 GB (Konic Optimized)

“The image shows a plate with two sandwiches, each cut in half and placed side by side. The sandwiches are made with toasted bread, and contain visible layers of lettuce, tomato, and ham. A slice of tomato is placed on the plate, and a piece of lettuce is positioned between the two sandwiches. The plate is set on a wooden table, and a knife is visible in the background.”

“What objects are in this image? List them.”

![COCO 2017 val image img_0050: a yellow propeller airplane in a gray sky](/images/research/lfm2.5-vl-3b-optimized/img_0050.jpg)

BASE · 6.25 GB BF16

“The image shows a single object: a small, single-engine propeller airplane. The airplane is the main focus of the image, and there are no other objects visible.”

FFN8192-INT4 · 2.22 GB (Konic Optimized)

“The image contains the following objects: 1. A yellow airplane 2. The airplane’s registration number (SP-AWF) 3. The airplane’s red-tipped tail 4. The airplane’s blue nose 5. The airplane’s propeller 6. The airplane’s landing gear (wheels)”

“Write a one-sentence caption for this image.”

![COCO 2017 val image img_0100: a young girl holding a tray of rolls near an oven](/images/research/lfm2.5-vl-3b-optimized/img_0100.jpg)

BASE · 6.25 GB BF16

“A girl holds a tray of dough rolls in front of a Candy dishwasher.”

FFN8192-INT4 · 2.22 GB (Konic Optimized)

“A young girl in the kitchen holding a tray of croissants with her mouth open in joy.”

### Text prompts — base vs the smallest tier

Same facts, shorter wording on the Konic Optimized tier — the expected generation profile of a distilled, quantized model.

Prompt

BASE · 6.25 GB BF16

FFN8192-INT4 · 2.22 GB (Konic Optimized)

What is the capital of France? Answer in one sentence.

“The capital of France is Paris.”

“Paris.”

Explain the difference between a cat and a dog in one sentence.

“Cats are solitary, independent hunters with retractable claws and a more aloof demeanor, whereas dogs are social, pack-oriented companions known for their loyalty and the ability to bark.”

“Cats are independent, solitary hunters, while dogs are social, pack-oriented animals.”

What would you pack for a beach vacation? List three items.

“Here are three essential items to pack for a beach vacation: 1. \*\*Sunscreen:\*\* Protecting your skin from harmful UV rays is crucial to prevent sunburn and long-term skin damage. 2. \*\*Beach Towel:\*\* A large, quick-dry towel is necessary for lounging, changing clothes, and drying off after swimming. 3. \*\*Swimsuit:\*\* Comfortable swimwear is required for enjoying the water and the sand.”

“Here are three essential items for a beach vacation: 1. \*\*Sunscreen\*\* – Protects your skin from harmful UV rays and prevents sunburn. 2. \*\*Beach towel\*\* – Provides a comfortable place to relax and dry off after swimming. 3. \*\*Swimsuit\*\* – Allows you to enjoy the water and feel cool.”

Recorded outputs, greedy decode, temperature 0, via the chat template.

### Tool calling — base vs the smallest tier

Five OpenAI-format tools (book\_flight, get\_weather, send\_email, set\_alarm, restaurant\_search), 10 rounds per model. Calls are the models’ native `<|tool_call_start|>[...]` format, markers elided; outputs verbatim.

Round

Prompt

BASE · 6.25 GB BF16

FFN8192-INT4 · 2.22 GB (Konic Optimized)

flight-simple

Book a flight from Paris to London tomorrow.

“book\_flight(origin="Paris", destination="London", date="2023-10-06")”

“book\_flight(origin="Paris", destination="London", date="2023-10-26")”

flight-complex

Book a flight from New York to Tokyo on 2026-09-15 in business class for two passengers: Alice and Bob.

“book\_flight(origin="New York", destination="Tokyo", date="2026-09-15", class="business", passengers=\[{'name': 'Alice'}, {'name': 'Bob'}\])”

“book\_flight(origin="New York", destination="Tokyo", date="2026-09-15", class="business", passengers=\[{'name': 'Alice'}, {'name': 'Bob'}\])”

weather-unit

What’s the weather in Berlin in Fahrenheit?

“get\_weather(city="Berlin", unit="fahrenheit")”

“get\_weather(city="Berlin", unit="fahrenheit")”

weather-simple

What’s the weather in Madrid?

“get\_weather(city="Madrid", unit="celsius")”

“get\_weather(city="Madrid", unit="celsius")”

email

Send a high priority email to john@example.com asking about the report deadline.

“send\_email(recipient="john@example.com", subject="Inquiry About Report Deadline", body='Hello John, I hope you are well. Could you please let me know the deadline for the report? Thank you.', priority="high")”

“send\_email(recipient="john@example.com", subject="Report Deadline", body="Please provide the report deadline.", priority="high")”

alarm

Set an alarm for 7:30 AM on weekdays.

“set\_alarm(time="07:30", days=\['Mon', 'Tue', 'Wed', 'Thu', 'Fri'\])”

“set\_alarm(time="07:30", days=\['Mon', 'Tue', 'Wed', 'Thu', 'Fri'\])”

restaurant

Find an Italian restaurant in Rome with outdoor seating under 50 euros.

“restaurant\_search(cuisine="Italian", city="Rome", max\_price=50, outdoor\_seating=True)”

“restaurant\_search(cuisine="Italian", city="Rome", max\_price=50, outdoor\_seating=True)”

multi-tool

What’s the weather in Barcelona and also set an alarm for 9 AM tomorrow?

“get\_weather(city="Barcelona", unit="celsius"), set\_alarm(time="09:00", days=\['Mon'\])”

“get\_weather(city="Barcelona", unit="celsius"), set\_alarm(time="09:00", days=\['Mon'\])”

no-tool-1

Hi! Tell me a fun fact about the ocean.

“get\_weather(city="Ocean City", unit="celsius") — over-triggered a tool call on a no-tool prompt”

“The ocean covers about 71% of Earth’s surface and contains over 95% of the planet’s water. It’s so vast that it’s difficult to imagine its depth or the number of species living in it! — no call”

no-tool-2

What is the capital of Japan?

“The capital of Japan is Tokyo. — no call”

“The capital of Japan is Tokyo. — no call”

Recorded outputs, greedy decode, temperature 0, max\_tokens 220. Tool selection and arguments match on all seven tool rounds; on the no-tool controls the base over-triggered once (replicated in two independent batteries) while the Konic Optimized model abstained correctly on both.

## Limitations

- **Dequant-eval caveat.** The harness evaluates exact dequant numerics, not quantized kernels; real-kernel evidence is the vLLM comparison batteries.
- **Heavy task benchmarks pending.** ScreenSpot-v2 (80.7), RefCOCO (87.9), MME/MMStar are vendor claims being reproduced at phase gates via lm-evaluation-harness — the quick suite is the frequent-loop instrument, not the final word.
- **Vision drop deferred.** Dropping the vision tower’s top-3 layers without projector retrain shifts the mm distribution measurably; shipping it requires the retrain (phase 3).
- **Single recorded runs.** Point estimates (plus bootstrap CIs on the harness axes) carry unknown single-run noise; tool batteries are 10 curated rounds.

## Reproducibility

All three tiers are public under `konic-labs` in the “LFM2.5 3B Vision - Optimized” collection (LFM Open License v1.0 retained). The pipeline is committed and runnable on one L4:

```bash
# 0. Build the baseline reference cache once (paired stats, ~47 s)
PYTHONPATH=code /data/venv/bin/python3 -m eval_harness.cli build-reference \
  --suite quick --baseline <bf16_dir> --tag lfm25-vl-3b-bf16

# 1. Prune: joint-SwiGLU Wanda, uniform tier (scores reusable across tiers)
python3 -m pruning.wanda_prune --baseline <bf16_dir> --out <pruned_dir> \
  --target 8192 --text-samples 256 --mm-samples 32

# 2. Recover: KL distillation to the cached baseline distribution (top-64)
python3 -m pruning.cache_teacher --baseline <bf16_dir> \
  --manifest text_train_disjoint_2048.jsonl --out teacher_disjoint2048_topk.npz
python3 -m pruning.lora_recover --model <pruned_dir> --out <rec_dir> \
  --distill teacher_disjoint2048_topk.npz \
  --train-manifest text_train_disjoint_2048.jsonl

# 3. Quantize: W4A16 GPTQ + W8 RTN + int8 embeddings
python3 -m pruning.quantize --model <rec_dir> --out <quant_dir> --calib-samples 128

# 4. Eval any candidate against the cache
PYTHONPATH=code /data/venv/bin/python3 -m eval_harness.cli run \
  --suite quick --candidate <candidate_dir> --candidate-tag <name>
```

Tested environment: NVIDIA L4 (23 GB, sm\_89), 4 vCPU / 15 GiB RAM, PyTorch 2.11.0+cu128, Transformers 5.14.1, vLLM 0.26.0 (flashinfer JIT with `FLASHINFER_DISABLE_VERSION_CHECK=1`), Datasets 5.0, Python 3.10.12. Training manifests are pinned JSONL and verified disjoint from the eval shard; teacher caches are committed alongside.

## Conclusion

The width-slack hypothesis is supported, and the surgical compression is near-lossless in practice: 24% FFN width removal with small distillation LoRA lands at 5.30 GB, the 33% cut at 4.92 GB, and hand-rolled W4A16 GPTQ + W8 RTN carries the 8192 tier to 1.93 GB packed (−69%) — native INT4/INT8 compressed-tensors at 2.22 GB (Konic Optimized). All three tiers sit at or below baseline shard likelihood (PPL ratios 0.79–0.85, gates pass), keep top-5 token agreement at 0.95–0.99, and the quantization pass adds only +0.02–0.06 nats over its BF16 parent. Live comparison on the smallest tier reproduces the same scenes, facts, and tool calls as the base — 10/10 tool rounds, including correct abstention on no-tool prompts where the base over-triggered.

Remaining work is the phase-gate heavy benchmarks (ScreenSpot-v2, RefCOCO, MME/MMStar), the deferred vision-drop retrain, and the FP8 vs W4A16 head-to-head. Teams evaluating these artifacts should measure on their own distributions rather than extrapolate from these recorded runs.

## Sources

- Collection: [konic-labs — LFM2.5 3B Vision - Optimized](https://huggingface.co/collections/konic-labs/lfm25-3b-vision-optimized-6a7df980f5dd03eeed96d4a1)
- Releases: [LFM2.5-VL-3B-ffn8192](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192) (5.30 GB) · [ffn7168](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn7168) (4.92 GB) · [ffn8192-int4](https://huggingface.co/konic-labs/LFM2.5-VL-3B-ffn8192-int4) (2.57 GB, compressed-tensors, vLLM-ready)
- Base model: `LiquidAI/LFM2.5-VL-3B` — 3.123B params, 22 conv + 8 attention layers, SigLIP2 so400m NaFlex vision tower; LFM Open License v1.0.
- Evaluation harness: `code/eval_harness/` in the lab repo — baseline reference cache + paired metrics, bit-identity control, block bootstrap CIs.
- Compression pipeline: `code/pruning/` —`wanda_prune.py`, `lora_recover.py`, `quantize.py` + `quant_loader.py`; compressed-tensors conversion `code/serve/to_ct_format.py`; comparison batteries `code/serve/compare_vllm.py` and `compare_tools2.py`.
- Prior art: Wanda (Sun et al., 2023); GPTQ (Frantar et al., 2022); LoRA (Hu et al., 2021); POP prefill-only pruning (arXiv 2602.03295); SigLIP2 NaFlex (google/siglip2-so400m-patch16-naflex).

Key takeaways

- **1.93 GB** — Packed W4A16/W8 artifact (from 6.25 GB, −69%)
- **0.79–0.85** — PPL ratios vs base — at or below baseline, gates PASS
- **0.95–0.99** — Top-5 token agreement with the base (both shards)
- **10 / 10** — Tool rounds on the smallest tier, INT4 (Konic Optimized)

---

Source: https://konic.io/research/lfm2.5-vl-3b-optimized