---
title: "LFM2.5 Encoder 230M + SigLIP2: A Compact Multimodal Encoder"
description: "We augment the 230M LFM2.5 Encoder with SigLIP2 and 32 soft tokens; GPTQ INT4 cuts 923 MB to 370 MB, down 59.9%, retaining 0.973 AUROC retrieval quality."
url: https://konic.io/research/lfm2.5-multimodal-encoder-230m
published: 2026-08-12
updated: 2026-08-12
section: "Insights"
publisher: "Konic Labs, Inc."
authors: ["Gokalp Katkat", "Ege Sabanci"]
keywords: ["LFM2.5 Encoder", "multimodal encoder", "SigLIP2", "image-text retrieval", "encoder-only model"]
about: ["Compact multimodal encoder", "SigLIP2 vision tower", "Encoder-only retrieval", "LFM2.5 Encoder"]
---

# LFM2.5 Encoder 230M + SigLIP2: A Compact Multimodal Encoder

> We augment the 230M LFM2.5 Encoder with SigLIP2 and 32 soft tokens; GPTQ INT4 cuts 923 MB to 370 MB, down 59.9%, retaining 0.973 AUROC retrieval quality.

*Konic Research · Insights · 12 min read*

## Artifacts
- [huggingface.co/konic-labs/LFM2.5-multimodal-encoder-230M](https://huggingface.co/konic-labs/LFM2.5-multimodal-encoder-230M)

## Introduction

Liquid AI’s **LFM2.5 Encoder 230M** is a compact bidirectional masked-language encoder: 229.7M parameters, hidden width 1,024, 14 layers, and a 65,536-row vocabulary. It is an excellent text representation model, but it is *not* natively multimodal — there is no vision tower in the checkpoint. This post records an external vision augmentation: we bolt a **SigLIP2 base vision encoder** onto the frozen LFM2.5 body through a learned 32-token soft-visual projector, adapt the stack for retrieval and image-text matching, and ship the result as a **GPTQ INT4 package at 370.46 MB** — 59.9% smaller than the original FP32 checkpoint package.

The claim is deliberately scoped. This is an **encoder-only** model: it produces image/text representations and image-text matching scores for retrieval, matching, and representation extraction. It does not generate captions or chat responses. The evaluation is **decoder-free and leakage-free**: for retrieval, the image representation is built from BOS + image marker + visual tokens only, and the candidate caption is never fed into the image branch.

These are **recorded run summaries** from the LFM2.5 encoder lab (reports under `reports/` in the lab repository), not general benchmarks.

## Architecture

The image path runs through three frozen-or-trainable stages. The vision tower is frozen, the projector is the only trainable bridge into the frozen text body, and the final blocks of the body carry LoRA adapters for task alignment.

```text
Image
  ↓
SigLIP2 base patch16 256            [batch, 256, 768]   ← frozen vision tower
  ↓
Learned 32-token projector          [batch, 32, 1024]   ← trainable bridge
  (+ modality embedding)
  ↓
LFM2.5 bidirectional encoder        [batch, seq, 1024]  ← frozen body + LoRA (layers 10–13)
  ↓
Image/text embedding  ·  image-text matching score
```

The image branch is `BOS + image marker + 32 visual tokens` inserted at position 1 of the text sequence. The text branch is a caption-only LFM2.5 encoding. During clean retrieval evaluation the two branches never share the candidate caption; the matching head is the only component that may receive image and candidate text together, which is correct for image-text matching.

## Dataset and training

Training uses a 100,000-pair MONET subset materialized from `jasperai/monet` (revision `baae102c4c96c6571f248b86784c67c5af4fd57a`) with a seeded streaming shuffle and quality filters: `source == laion`, `aesthetic_jasperai > 0.5`, `nsfw_jasperai < 0.1`. An exact-byte audit found 0 duplicate images and 0 cross-split duplicates.

*Seeded stream split (seed 42). The calibration holdout is reserved for fused multimodal quantization calibration only.*

| Split | Pairs |
| --- | --- |
| Train | 73,000 |
| Calibration holdout (reserved, unused for weights) | 2,000 |
| Validation | 12,500 |
| Test (held out) | 12,500 |

Training is two stages with different frozen/trainable splits. Projector alignment freezes the vision tower and the entire LFM2.5 body; task adaptation then trains LoRA on the final blocks plus the matching head:

- **Projector alignment** — symmetric InfoNCE (temperature 0.07), 1,000 steps, batch 8, learning rate `1e-4`, FP32, seed 42. Initial loss 2.08 → 0.15.
- **Task adaptation** — retrieval InfoNCE plus binary image-text matching loss (weight 0.5), with LoRA on the final four LFM2.5 blocks (layers 10–13, rank 8 / alpha 16, 24 Linear modules), 1,000 steps, learning rate `5e-5`.

The adapted weights live in a separate task adapter (projector + LoRA + matching head); the original checkpoint is never modified.

## A leakage correction that changed the evaluation

The first training branch extracted the image representation from a **fused image + positive-caption** sequence. That is valid for image-text matching, but invalid for image-to-text retrieval — the image branch had already seen the target caption. A 2,048-pair diagnostic with an image-only representation exposed the problem: retrieval collapsed to near chance.

*Recorded diagnostic. The paired-caption adapter’s earlier high retrieval numbers were caption-leakage artifacts; the clean row is the full 12,500-pair BF16 evaluation.*

| Branch | Pairs | Image→text R@1 | Text→image R@1 | Hard-negative AUROC |
| --- | --- | --- | --- | --- |
| Original paired-caption branch | 2,048 | 0.0024 | 0.0000 | 0.5420 |
| Clean retrained branch (image-only representation) | 12,500 | 0.1194 | 0.1578 | 0.4974 |

The clean branch retrains with an image-only representation for contrastive alignment and retrieval — the caption is never sent through the image branch — while the matching loss still uses the fused image-plus-caption representation, which is correct for ITM. This clean branch is the publication branch, and the earlier paired-caption artifact is retained only as a leakage diagnostic.

## GPTQ INT4 with fused multimodal calibration

The merged LoRA-adapted body is compressed with GPTQ INT4 (W4A16). The calibration pass is **fused**: it uses 256 samples of actual projected visual tokens plus caption embeddings — not text-only inputs — so the quantization reflects the real multimodal path.

*Recorded run summary. The overlay used compressed-tensors 0.17.2a20260729 with the lab’s LFM2-specific SwiGLU mapping.*

| Compression setting | Value |
| --- | --- |
| Method | GPTQ INT4 |
| Weight / activation format | W4A16 |
| Quantized layers | Linear layers; lm_head excluded |
| Block size | 128 |
| Activation order / dampening | Static / 0.01 |
| Calibration | 256 fused multimodal inputs_embeds samples |
| Max calibration sequence length | 160 tokens |
| Compressor revision | 8cec0acc1931de6f8f73257151ab7007c14dbf4e |

*Recorded artifact sizes. The GPTQ release is 59.9% smaller than the original package and 22.0% smaller than the clean BF16 package. The external SigLIP2 vision tower is downloaded separately and is not included.*

| Artifact | Model weights | Full package | Relative to original |
| --- | --- | --- | --- |
| Original LFM2.5 Encoder 230M FP32 | 918.79 MB | 923.65 MB | 100% |
| Clean BF16 reference | 459.40 MB | 474.81 MB | 51.4% |
| Clean GPTQ INT4 release | 355.03 MB | 370.46 MB | 40.1% |

*Text smoke prompts; hidden/logit cosine is a secondary diagnostic, task quality is primary.*

| Fidelity metric | GPTQ INT4 vs BF16 |
| --- | --- |
| Hidden-state cosine | 0.9427 |
| Logit cosine | 0.9654 |
| Logit top-1 agreement | 0.8485 |

> **Runtime role**
>
>
>
> On the NVIDIA L4 used here, the current generic compressed-tensors backend makes GPTQ INT4 *slower* than native BF16 — fused multimodal batch-4 mean 44.62 ms vs 17.40 ms, peak VRAM 2.38 GiB vs 1.21 GiB. The INT4 selection is size-oriented; native packed kernels or a custom backend would be required to realize a runtime benefit.

## Evaluation

All retrieval below uses the clean image-only representation, on the full 12,500-pair MONET test split (bootstrap 95% CIs over pair-level ranks) plus a bounded 2,000-example external CC3M subset. The 12,500-way random R@1 baseline is 0.00008; the 2,000-way baseline is 0.0005.

*Clean retrieval, recorded run. GPTQ INT4 retains ~91% of the BF16 image→text R@1 on MONET; CC3M shows expected domain degradation but stays far above random.*

| Model | Image→text R@1 (95% CI) | Text→image R@1 (95% CI) | i→t R@5 | t→i R@5 | Mean rank i→t / t→i |
| --- | --- | --- | --- | --- | --- |
| MONET BF16 reference | 0.1194 [0.1134, 0.1244] | 0.1578 [0.1515, 0.1632] | 0.3089 | 0.3598 | 58.66 / 50.04 |
| MONET GPTQ INT4 | 0.1091 [0.1036, 0.1147] | 0.1460 [0.1399, 0.1511] | 0.2959 | 0.3418 | 63.38 / 53.78 |
| CC3M external, GPTQ INT4 | 0.0865 [0.0735, 0.0995] | 0.1145 [0.1010, 0.1310] | 0.2190 | 0.2835 | 137.34 / 112.76 |

**Matching** is evaluated against two negative families: cyclic in-batch negatives (easy) and text-nearest hard negatives chosen by independent text-only embedding similarity (hard).

*Recorded run. The head recognizes coarse compatibility but does not reliably resolve semantically close captions — an important limitation.*

| Model | Cyclic AUROC | Cyclic AUPRC | Hard AUROC | Hard AUPRC | Hard positive wins |
| --- | --- | --- | --- | --- | --- |
| MONET BF16 | 0.9747 | 0.9634 | 0.4974 | 0.4974 | 0.4861 |
| MONET GPTQ INT4 | 0.9733 | 0.9619 | 0.4961 | 0.4974 | 0.4914 |
| CC3M external GPTQ | 0.8065 | 0.7932 | 0.5256 | 0.5217 | 0.5665 |

**Visual ablation** on 512 MONET pairs swaps the visual input to isolate how much of the image representation comes from the learned tokens.

*Recorded run, 512 pairs. Cosine to the trained image representation: zero 0.0094, random 0.0247, shuffle 0.1111 (BF16).*

| Visual input | BF16 i→t R@1 | BF16 t→i R@1 | GPTQ i→t R@1 | GPTQ t→i R@1 |
| --- | --- | --- | --- | --- |
| Trained visual tokens | 0.4922 | 0.5449 | 0.4883 | 0.5352 |
| Zero visual tokens | 0.0020 | 0.0000 | 0.0020 | 0.0000 |
| Random visual tokens | 0.0020 | 0.0000 | 0.0020 | 0.0000 |
| Shuffled visual tokens | 0.0117 | 0.0195 | 0.0117 | 0.0215 |
| No-image (fixed zero control) | 0.0020 | 0.0000 | 0.0020 | 0.0000 |

> **The visual stitch is doing the work**
>
>
>
> Replacing the trained visual tokens with zero, random, or shuffled tokens collapses retrieval to chance, and shuffling token order destroys it. This is the strongest evidence that the learned soft-visual tokens contribute to the image representation rather than passing through a text shortcut.

**Robustness** on 256 MONET examples: retrieval stays well above random under blur, grayscale, darkening, and center-crop perturbations (GPTQ image→text R@1: 0.5938 / 0.5742 / 0.6367 / 0.5391; BF16 similar). **Image-conditioned masked-token prediction** does not beat zero-image or text-only controls (BF16 trained loss 2.20 vs 1.90 zero-image vs 2.00 text-only), so MLM is not yet evidence of fine-grained visual grounding.

## Limitations

- **Encoder-only.** No generation, no decoder, no captioning. The model is for retrieval, matching, and representation learning.
- **Hard negatives are near chance.** Text-nearest hard-negative AUROC is ~0.50 on MONET and ~0.53 on CC3M; cyclic negatives (AUROC 0.9733) are substantially easier.
- **No fine-grained grounding signal yet.** Image-conditioned masked-token prediction does not outperform zero-image/text-only controls on the current caption data.
- **Bounded external evaluation.** External generalization uses a 2,000-example CC3M subset, not full COCO / Flickr30k / Winoground (gated or legacy-script blocked); its source terms require review before redistribution.
- **Duplicate audit is exact-byte only.** Perceptual duplicates are not covered.
- **Size-oriented compression.** On the tested L4 the generic compressed-tensors backend makes INT4 slower and more VRAM-hungry than BF16; the release targets artifact size.
- **Single recorded runs.** Point estimates except retrieval CIs carry unknown single-run noise.

## Reproducibility

The compact release is on Hugging Face at [konic-labs/LFM2.5-multimodal-encoder-230M](https://huggingface.co/konic-labs/LFM2.5-multimodal-encoder-230M), with the GPTQ weights, the bidirectional runtime implementation, the task adapter (projector + matching head), the quantization recipe, and a saved `evaluation/` directory containing the BF16, GPTQ, and external CC3M metrics. Every step — dataset preparation, projector and task training, compression, structural surgery, and the deep evaluation harness — is reproducible end-to-end, with step-by-step reports under `reports/` and `reports/deep_eval/`.

One deterministic example:

```bash
python run_multimodal.py \
  --model . \
  --image /path/to/image.jpg \
  --text "A photograph of a small red vehicle beside a building." \
  --device cuda:0 --dtype float32
```

Tested environment: NVIDIA L4 (23,034 MiB), PyTorch 2.11.0+cu128, Transformers 5.14.1, Datasets 5.0.1, llm-compressor at revision `8cec0acc`, compressed-tensors 0.17.2a20260729 overlay.

## Conclusion

A 230M bidirectional text encoder can be turned into a useful multimodal encoder without touching the original checkpoint: a frozen SigLIP2 tower, 32 learned soft tokens, and LoRA adaptation produce image/text representations that retrieve held-out MONET pairs far above random, survive moderate visual perturbations, and depend on the actual visual tokens (ablations collapse without them). GPTQ INT4 with fused multimodal calibration keeps ~91% of the BF16 retrieval R@1 in a package 59.9% smaller than the original FP32 checkpoint.

The honest boundary is equally important: this is an encoder for retrieval and matching, not a generator; fine-grained hard-negative discrimination and visual grounding are open problems; and the quantized artifact is a size win, not yet a runtime win. Teams considering this stack should measure on their own distribution rather than extrapolate from these recorded runs.

## Sources

- Hugging Face release: [konic-labs/LFM2.5-multimodal-encoder-230M](https://huggingface.co/konic-labs/LFM2.5-multimodal-encoder-230M) — GPTQ INT4 weights, task adapter, recipe, and saved metrics.
- Base model: Liquid LFM2.5 Encoder 230M — 229,693,184 parameters, bidirectional masked-language encoder, hidden width 1,024, 14 layers.
- Vision tower: `google/siglip2-base-patch16-256` — 256 patch tokens at 768 width, projected to 32 × 1,024.
- Dataset: MONET 100k subset of `jasperai/monet` (revision `baae102c`), filters `source == laion`, `aesthetic_jasperai > 0.5`, `nsfw_jasperai < 0.1`.
- External subset: 2,000 bounded `pixparse/cc3m-wds` examples; source terms require review before redistribution.
- Compression: GPTQ INT4 W4A16 via llm-compressor revision `8cec0acc`, 256 fused multimodal calibration samples.

Key takeaways

- **370 MB** — GPTQ INT4 release (from 923.65 MB, −59.9%)
- **0.1194** — Clean image→text R@1, BF16, 12,500 MONET pairs
- **0.1091** — GPTQ INT4 image→text R@1 — ~91% retained
- **0.9733** — Cyclic-negative matching AUROC (hard negatives: near chance)

---

Source: https://konic.io/research/lfm2.5-multimodal-encoder-230m