From-Scratch AWQ INT4 Quantization on Qwen3-8B

Activation-aware Weight Quantization (AWQ, Lin et al. 2023) is the de-facto method for turning a 7–8B-class FP16 LLM into a deployable INT4 model. It protects the weight channels aligned with large-magnitude activations by scaling them up before quantization, then compensating the activations. The production toolchain, however, is largely black-box - AutoAWQ and GPTQ packages that hide the scale search, the packing, and the runtime handoff behind a single call.

We wanted the opposite: a from-scratch, pure-PyTorch, model-agnostic implementation where every stage - activation statistics, per-layer scale search, INT4 packing, reconstruction verification, and the bridge to a real INT4 runtime - is inspectable. And we wanted to prove, end-to-end on a viable-sized model on CUDA, that it produces a 4× smaller model with near-zero loss.

This post presents that implementation and its validation on Qwen3-8B. The pipeline produces an INT4 model that is 4.0× smaller in linear weights and runs at 1.034× FP16 perplexity in a real INT4 GEMM runtime, with coherent greedy generation indistinguishable in substance from the FP16 baseline.

AWQ scaling

For a linear layer with weight $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and per-input-channel activation magnitude $x_{\max} = |X|.\text{mean}(0)$, AWQ chooses a per-channel scale

scales the weights up by $s$ before quantization ($W' = W \cdot s$), and divides the activations by $s$ at inference. The reference implementation (mit-han-lab/llm-awq) folds $1/s$ into the preceding norm's weight ($\text{norm.weight} /= s$) rather than storing $s$ per linear. The optimal $\alpha$ is found by grid search, scored by the activation-weighted reconstruction error $\| (W - Q(W \cdot s)/s) \cdot \text{diag}(x_{\max}) \|^{2}$;$\alpha = 0$ gives $s = 1$ (plain RTN), so the search can only improve on RTN, never worsen it.

Group-wise symmetric INT4

Weights are quantized in groups of group_size input channels with one FP16 scale per group: $q_{\text{scale}} = \max|\text{group}| / 7$, $q = \text{round}(w / q_{\text{scale}}).\text{clamp}(-7, 7)$, so the signed grid is $[-7, 7]$ (15 codes). Two INT4 are packed per byte in the internal artifact.

AutoAWQ / HF-AWQ GEMM on-disk format

The runtime-loadable layout is unsigned $[0, 15]$ int4 packed 8-per-int32, with an AWQ pack order $[0,2,4,6,1,3,5,7]$ governing the within-int32 ordering. Reconciling our per-linear $s$ with this shared-norm convention is the hard part - covered in §4.

The pipeline is deliberately linear and inspectable. Each phase is independently runnable and produces a discrete artifact on disk.

FP16 model  ──► calibrate ──► calibration_stats.pt ──► scales ──► awq_scales.pt
                                                                     │
                                                                     ▼
prompts ──────────────────────────────────────────────────► quantize ──► quantized_state.pt + metadata.json
                                                                              │
                                                                              ▼
                                                                       verify (MSE)
                                                                              │
                                                                              ▼
                                                                       export ──► awq_hf/ (AutoAWQ GEMM)
                                                                                       qweight / qzeros / scales
                                                                                       + folded-norm FP16

The hard part is producing a model a real INT4 runtime can load, because our per-linear $s$ does not map 1:1 onto the standard AWQ runtime convention:

• q_proj / k_proj / v_proj share input_layernorm;
• gate_proj / up_proj share post_attention_layernorm;
• you cannot fold three different $s$ into one norm;
• o_proj and down_proj have no preceding norm at all.

The exporter reconciles this as follows:

1. Aggregate the per-linear $s$ into one shared scale per norm group (geometric mean over the member linears) and fold $1/s$ into that norm ($\text{norm.weight} /= s_{\text{shared}}$).
2. Re-quantize the 5 norm-following linears (q/k/v/gate/up) with the shared $s$.
3. Export o_proj / down_proj as plain RTN INT4 ($s = 1$) - they have no preceding norm to absorb a scale.
4. Map the signed $[-7, 7]$grid onto AutoAWQ's unsigned $[0, 15]$ grid by storing $\text{zero} = 7$ and $q_{\text{unsigned}} = q_{\text{signed}} + 7$, so the runtime's $(q - \text{zero}) \cdot \text{scale}$ exactly reproduces our group dequant $q_{\text{signed}} \cdot \text{group\_scale}$ - zero re-quantization error on the AWQ-scaled linears.

The output directory contains AutoAWQ GEMM qweight / qzeros / scales per quantized linear, the FP16 (folded) norms + embeddings + lm_head, and a quantize_config.json. AutoAWQ loads it directly and replaces each quantized linear with a WQLinear INT4 GEMM module.

On the production-relevant 8B model, the exported AWQ-INT4 artifact is 4.0× smaller in linear weights (13.9 GB → 3.5 GB) and runs at 1.034× FP16 perplexity(9.75 → 10.08) in AutoAWQ's real INT4 GEMM runtime - comfortably under the 1.10× pass bar, with coherent greedy generation across every prompt.

The single most important design choice is how the AWQ scale $s$ is applied at inference. There are two strategies:

• Weight-dequantization: dequantize each linear to $Q(W \cdot s)/s$ as an FP16 weight and run a normal FP16 forward (no norm change).
• Norm-folding: store $Q(W \cdot s)$ and fold $1/s$ into the preceding norm ($\text{norm.weight} /= s$), so the activation is divided by $s$. This is the reference mit-han-lab/llm-awq convention and what awq export does.

They are not equivalent. On a single 1.7B projection layer:

Grid search wins on per-layer weight reconstruction (~2× lower MSE), confirming the search is real and the scoring function is correctly weighted. Fixed-$\alpha$ wins on end-to-end runtime PPL - and this is not a scoring bug.

• The win, restated. A from-scratch AWQ implementation produces a 4× smaller, +3.4%-PPL Qwen3-8B that runs in a real INT4 GEMM runtime - production-relevant, not a toy result on a sub-viable model.
• RTN for o_proj / down_proj. These two linears per block have no preceding norm, so the exporter leaves them at RTN ($s = 1$) rather than applying a scale it cannot fold. A follow-up can absorb their $s$ into the preceding projection (v_proj / up_proj output channels) for further quality headroom.
• Norm-group-level search. The ablation shows per-layer grid search optimizes the wrong unit for the exportable format; norm-group search is the natural next improvement.
• Generalization. Evaluated on Qwen3; the model.layers.{i}.* convention makes Llama/Mistral support straightforward. lm_head is intentionally skipped (vocabulary projection is too sensitive to INT4).

A from-scratch, pure-PyTorch, model-agnostic AWQ implementation, with a runtime-loadable export, delivers a 4× smaller Qwen3-8B at 1.034× FP16 perplexity - near-zero loss, coherent generation, runnable in a real INT4 GEMM runtime. The decisive factor is norm-folding the AWQ scale (20× more accurate per weight than weight-dequantization), and the clear next step is norm-group-level scale search to bring smaller models into the same near-lossless regime.

The implementation is open-source at github.com/egesabanci/awq, with the full pipeline (calibrate / scales / quantize / verify / export) and a standalone evaluator for perplexity and generation.

All numbers here are regenerated by the commands below.

# Quantize (CUDA)
awq run --model /data/models/Qwen3-8B --dataset wikitext \
  --output-dir /data/out-8b --samples 128 --max-length 2048 \
  --device cuda --quantize-strategy all --group-size 128 --verify-layers 5

# Export to a runtime-loadable AutoAWQ/HF-AWQ model
awq export --model /data/models/Qwen3-8B \
  --from /data/out-8b/awq_quantized/quantized_state.pt \
  --to /data/out-8b/awq_hf --group-size 128 --device cuda

# Evaluate: FP16 baseline vs AWQ real-INT4 runtime
python eval/ppl.py --model /data/models/Qwen3-8B --config fp16
python eval/ppl.py --model /data/models/Qwen3-8B --config awq-runtime \
  --export-dir /data/out-8b/awq_hf --gen