LFM2.5 Encoder 230M + SigLIP2: A Compact Multimodal Encoder

Liquid AI’s LFM2.5 Encoder 230M is a compact bidirectional masked-language encoder: 229.7M parameters, hidden width 1,024, 14 layers, and a 65,536-row vocabulary. It is an excellent text representation model, but it is not natively multimodal — there is no vision tower in the checkpoint. This post records an external vision augmentation: we bolt a SigLIP2 base vision encoder onto the frozen LFM2.5 body through a learned 32-token soft-visual projector, adapt the stack for retrieval and image-text matching, and ship the result as a GPTQ INT4 package at 370.46 MB — 59.9% smaller than the original FP32 checkpoint package.

The claim is deliberately scoped. This is an encoder-only model: it produces image/text representations and image-text matching scores for retrieval, matching, and representation extraction. It does not generate captions or chat responses. The evaluation is decoder-free and leakage-free: for retrieval, the image representation is built from BOS + image marker + visual tokens only, and the candidate caption is never fed into the image branch.

These are recorded run summaries from the LFM2.5 encoder lab (reports under reports/ in the lab repository), not general benchmarks.

The image path runs through three frozen-or-trainable stages. The vision tower is frozen, the projector is the only trainable bridge into the frozen text body, and the final blocks of the body carry LoRA adapters for task alignment.

Image
  ↓
SigLIP2 base patch16 256            [batch, 256, 768]   ← frozen vision tower
  ↓
Learned 32-token projector          [batch, 32, 1024]   ← trainable bridge
  (+ modality embedding)
  ↓
LFM2.5 bidirectional encoder        [batch, seq, 1024]  ← frozen body + LoRA (layers 10–13)
  ↓
Image/text embedding  ·  image-text matching score

The image branch is BOS + image marker + 32 visual tokens inserted at position 1 of the text sequence. The text branch is a caption-only LFM2.5 encoding. During clean retrieval evaluation the two branches never share the candidate caption; the matching head is the only component that may receive image and candidate text together, which is correct for image-text matching.

Training uses a 100,000-pair MONET subset materialized from jasperai/monet (revision baae102c4c96c6571f248b86784c67c5af4fd57a) with a seeded streaming shuffle and quality filters: source == laion, aesthetic_jasperai > 0.5, nsfw_jasperai < 0.1. An exact-byte audit found 0 duplicate images and 0 cross-split duplicates.

Training is two stages with different frozen/trainable splits. Projector alignment freezes the vision tower and the entire LFM2.5 body; task adaptation then trains LoRA on the final blocks plus the matching head:

• Projector alignment — symmetric InfoNCE (temperature 0.07), 1,000 steps, batch 8, learning rate 1e-4, FP32, seed 42. Initial loss 2.08 → 0.15.
• Task adaptation — retrieval InfoNCE plus binary image-text matching loss (weight 0.5), with LoRA on the final four LFM2.5 blocks (layers 10–13, rank 8 / alpha 16, 24 Linear modules), 1,000 steps, learning rate 5e-5.

The adapted weights live in a separate task adapter (projector + LoRA + matching head); the original checkpoint is never modified.

The first training branch extracted the image representation from a fused image + positive-caption sequence. That is valid for image-text matching, but invalid for image-to-text retrieval — the image branch had already seen the target caption. A 2,048-pair diagnostic with an image-only representation exposed the problem: retrieval collapsed to near chance.

The clean branch retrains with an image-only representation for contrastive alignment and retrieval — the caption is never sent through the image branch — while the matching loss still uses the fused image-plus-caption representation, which is correct for ITM. This clean branch is the publication branch, and the earlier paired-caption artifact is retained only as a leakage diagnostic.

The merged LoRA-adapted body is compressed with GPTQ INT4 (W4A16). The calibration pass is fused: it uses 256 samples of actual projected visual tokens plus caption embeddings — not text-only inputs — so the quantization reflects the real multimodal path.

All retrieval below uses the clean image-only representation, on the full 12,500-pair MONET test split (bootstrap 95% CIs over pair-level ranks) plus a bounded 2,000-example external CC3M subset. The 12,500-way random R@1 baseline is 0.00008; the 2,000-way baseline is 0.0005.

Matching is evaluated against two negative families: cyclic in-batch negatives (easy) and text-nearest hard negatives chosen by independent text-only embedding similarity (hard).

Visual ablation on 512 MONET pairs swaps the visual input to isolate how much of the image representation comes from the learned tokens.

Robustness on 256 MONET examples: retrieval stays well above random under blur, grayscale, darkening, and center-crop perturbations (GPTQ image→text R@1: 0.5938 / 0.5742 / 0.6367 / 0.5391; BF16 similar). Image-conditioned masked-token prediction does not beat zero-image or text-only controls (BF16 trained loss 2.20 vs 1.90 zero-image vs 2.00 text-only), so MLM is not yet evidence of fine-grained visual grounding.

• Encoder-only. No generation, no decoder, no captioning. The model is for retrieval, matching, and representation learning.
• Hard negatives are near chance. Text-nearest hard-negative AUROC is ~0.50 on MONET and ~0.53 on CC3M; cyclic negatives (AUROC 0.9733) are substantially easier.
• No fine-grained grounding signal yet. Image-conditioned masked-token prediction does not outperform zero-image/text-only controls on the current caption data.
• Bounded external evaluation. External generalization uses a 2,000-example CC3M subset, not full COCO / Flickr30k / Winoground (gated or legacy-script blocked); its source terms require review before redistribution.
• Duplicate audit is exact-byte only. Perceptual duplicates are not covered.
• Size-oriented compression. On the tested L4 the generic compressed-tensors backend makes INT4 slower and more VRAM-hungry than BF16; the release targets artifact size.
• Single recorded runs. Point estimates except retrieval CIs carry unknown single-run noise.

The compact release is on Hugging Face at konic-labs/LFM2.5-multimodal-encoder-230M, with the GPTQ weights, the bidirectional runtime implementation, the task adapter (projector + matching head), the quantization recipe, and a saved evaluation/ directory containing the BF16, GPTQ, and external CC3M metrics. Every step — dataset preparation, projector and task training, compression, structural surgery, and the deep evaluation harness — is reproducible end-to-end, with step-by-step reports under reports/ and reports/deep_eval/.

One deterministic example:

python run_multimodal.py \
  --model . \
  --image /path/to/image.jpg \
  --text "A photograph of a small red vehicle beside a building." \
  --device cuda:0 --dtype float32

Tested environment: NVIDIA L4 (23,034 MiB), PyTorch 2.11.0+cu128, Transformers 5.14.1, Datasets 5.0.1, llm-compressor at revision 8cec0acc, compressed-tensors 0.17.2a20260729 overlay.

A 230M bidirectional text encoder can be turned into a useful multimodal encoder without touching the original checkpoint: a frozen SigLIP2 tower, 32 learned soft tokens, and LoRA adaptation produce image/text representations that retrieve held-out MONET pairs far above random, survive moderate visual perturbations, and depend on the actual visual tokens (ablations collapse without them). GPTQ INT4 with fused multimodal calibration keeps ~91% of the BF16 retrieval R@1 in a package 59.9% smaller than the original FP32 checkpoint.

The honest boundary is equally important: this is an encoder for retrieval and matching, not a generator; fine-grained hard-negative discrimination and visual grounding are open problems; and the quantized artifact is a size win, not yet a runtime win. Teams considering this stack should measure on their own distribution rather than extrapolate from these recorded runs.

• Hugging Face release: konic-labs/LFM2.5-multimodal-encoder-230M — GPTQ INT4 weights, task adapter, recipe, and saved metrics.
• Base model: Liquid LFM2.5 Encoder 230M — 229,693,184 parameters, bidirectional masked-language encoder, hidden width 1,024, 14 layers.
• Vision tower: google/siglip2-base-patch16-256 — 256 patch tokens at 768 width, projected to 32 × 1,024.
• Dataset: MONET 100k subset of jasperai/monet (revision baae102c), filters source == laion, aesthetic_jasperai > 0.5, nsfw_jasperai < 0.1.
• External subset: 2,000 bounded pixparse/cc3m-wds examples; source terms require review before redistribution.
• Compression: GPTQ INT4 W4A16 via llm-compressor revision 8cec0acc, 256 fused multimodal calibration samples.