---
title: "From-Scratch AWQ INT4 Quantization on Qwen3-8B"
description: "Our from-scratch PyTorch AWQ makes Qwen3-8B 4.0x smaller (13.9 GB to 3.5 GB) at just 1.034x FP16 perplexity on WikiText-2 via norm-folded INT4 scaling."
url: https://konic.io/research/awq-int4-qwen3-8b
published: 2026-07-03
updated: 2026-07-03
section: "Insights"
publisher: "Konic Labs, Inc."
authors: ["Gokalp Katkat", "Ege Sabanci"]
keywords: ["AWQ", "INT4 quantization", "Qwen3-8B", "from-scratch quantization", "weight quantization"]
about: ["AWQ quantization", "INT4 quantization", "Qwen3-8B optimization", "Weight quantization"]
---

# From-Scratch AWQ INT4 Quantization on Qwen3-8B

> Our from-scratch PyTorch AWQ makes Qwen3-8B 4.0x smaller (13.9 GB to 3.5 GB) at just 1.034x FP16 perplexity on WikiText-2 via norm-folded INT4 scaling.

*Konic Research · Insights · 12 min read*

## Artifacts
- [github.com/egesabanci/awq](https://github.com/egesabanci/awq)

## Introduction

Activation-aware Weight Quantization (AWQ, Lin et al. 2023) is the de-facto method for turning a 7–8B-class FP16 LLM into a deployable INT4 model. It protects the weight channels aligned with large-magnitude activations by scaling them up before quantization, then compensating the activations. The production toolchain, however, is largely black-box - AutoAWQ and GPTQ packages that hide the scale search, the packing, and the runtime handoff behind a single call.

We wanted the opposite: a **from-scratch, pure-PyTorch, model-agnostic** implementation where every stage - activation statistics, per-layer scale search, INT4 packing, reconstruction verification, and the bridge to a real INT4 runtime - is inspectable. And we wanted to prove, end-to-end on a viable-sized model on CUDA, that it produces a **4× smaller model with near-zero loss**.

This post presents that implementation and its validation on **Qwen3-8B**. The pipeline produces an INT4 model that is **4.0× smaller** in linear weights and runs at **1.034× FP16 perplexity** in a real INT4 GEMM runtime, with coherent greedy generation indistinguishable in substance from the FP16 baseline.

## Background

### AWQ scaling

For a linear layer with weight $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ and per-input-channel activation magnitude $x_{\max} = |X|.\text{mean}(0)$, AWQ chooses a per-channel scale

$$
s = \frac{x_{\max}^{\alpha}}{\text{mean}(x_{\max})^{\alpha}} \qquad \alpha \in [0, 1)
$$

scales the weights **up** by $s$ before quantization ($W' = W \cdot s$), and divides the **activations** by $s$ at inference. The reference implementation (`mit-han-lab/llm-awq`) folds $1/s$ into the **preceding norm**'s weight ($\text{norm.weight} /= s$) rather than storing $s$ per linear. The optimal $\alpha$ is found by grid search, scored by the activation-weighted reconstruction error $\| (W - Q(W \cdot s)/s) \cdot \text{diag}(x_{\max}) \|^{2}$;$\alpha = 0$ gives $s = 1$ (plain RTN), so the search can only improve on RTN, never worsen it.

### Group-wise symmetric INT4

Weights are quantized in groups of `group_size` input channels with one FP16 scale per group: $q_{\text{scale}} = \max|\text{group}| / 7$, $q = \text{round}(w / q_{\text{scale}}).\text{clamp}(-7, 7)$, so the signed grid is $[-7, 7]$ (15 codes). Two INT4 are packed per byte in the internal artifact.

### AutoAWQ / HF-AWQ GEMM on-disk format

The runtime-loadable layout is unsigned $[0, 15]$ int4 packed 8-per-int32, with an AWQ pack order $[0,2,4,6,1,3,5,7]$ governing the within-int32 ordering. Reconciling our per-linear $s$ with this shared-norm convention is the hard part - covered in §4.

## Method

The pipeline is deliberately linear and inspectable. Each phase is independently runnable and produces a discrete artifact on disk.

```text
FP16 model  ──► calibrate ──► calibration_stats.pt ──► scales ──► awq_scales.pt
                                                                     │
                                                                     ▼
prompts ──────────────────────────────────────────────────► quantize ──► quantized_state.pt + metadata.json
                                                                              │
                                                                              ▼
                                                                       verify (MSE)
                                                                              │
                                                                              ▼
                                                                       export ──► awq_hf/ (AutoAWQ GEMM)
                                                                                       qweight / qzeros / scales
                                                                                       + folded-norm FP16
```

### Calibrate

A forward hook on every `nn.Linear` aggregates the per-input-channel activation magnitude **on-the-fly** as a running sum: $c_i = x.\text{abs}().\text{mean}(0)$, accumulated per layer with a running count. Memory is $O(d_{\text{in}})$ per layer - not $O(\text{samples} \times \text{tokens} \times d_{\text{in}})$ - so hundreds of calibration samples cost nothing beyond a single forward. We calibrate on 128 WikiText-2 samples at `max_length = 2048`.

### Scales

For each linear, we grid-search $\alpha$ over `n_grid = 20` values in $[0, 1)$ and score each candidate $s$ by **actually quantizing** the weight with it (the pseudo-quantizer mirrors the real quantizer's grid exactly) and computing $\| (W - Q(W \cdot s)/s) \cdot \text{diag}(x_{\max}) \|^{2}$. The layer count for depth-dependent skip strategies is **derived from the calibration stats** via `model.layers.{i}.\*`, not hardcoded - the implementation is model-agnostic for any Llama/Qwen/Mistral-style causal LM.

### Quantize

The quantizer **never loads the full model**. It streams `safetensors` one weight tensor at a time, applies $W \cdot s$, performs group-wise symmetric INT4 quantization, packs two INT4 per byte, and writes `quantized_state.pt` + `metadata.json`. Peak memory is one tensor plus its packed output. All 252 linear layers of Qwen3-8B are quantized this way.

### Verify

The canonical dequantizer inverts the packer and reports MSE against the original FP16 weight read from disk. This is a reconstruction-quality check on the quantization itself, independent of any runtime.

## Export - the design behind near-zero loss

The hard part is producing a model a real INT4 runtime can load, because our per-linear $s$ does not map 1:1 onto the standard AWQ runtime convention:

- `q_proj` / `k_proj` / `v_proj` share `input_layernorm`;
- `gate_proj` / `up_proj` share `post_attention_layernorm`;
- you cannot fold three different $s$ into one norm;
- `o_proj` and `down_proj` have **no preceding norm** at all.

The exporter reconciles this as follows:

1. **Aggregate** the per-linear $s$ into one **shared scale per norm group** (geometric mean over the member linears) and **fold** $1/s$ into that norm ($\text{norm.weight} /= s_{\text{shared}}$).
2. **Re-quantize** the 5 norm-following linears (`q`/`k`/`v`/`gate`/`up`) with the shared $s$.
3. **Export `o_proj` / `down_proj` as plain RTN INT4** ($s = 1$) - they have no preceding norm to absorb a scale.
4. Map the signed $[-7, 7]$ grid onto AutoAWQ's unsigned $[0, 15]$ grid by storing $\text{zero} = 7$ and $q_{\text{unsigned}} = q_{\text{signed}} + 7$, so the runtime's $(q - \text{zero}) \cdot \text{scale}$ **exactly reproduces** our group dequant $q_{\text{signed}} \cdot \text{group\_scale}$ - **zero re-quantization error** on the AWQ-scaled linears.

The output directory contains AutoAWQ GEMM `qweight` / `qzeros` / `scales` per quantized linear, the FP16 (folded) norms + embeddings + `lm_head`, and a `quantize_config.json`. AutoAWQ loads it directly and replaces each quantized linear with a `WQLinear` INT4 GEMM module.

## Experimental setup

*The pass bar for near-zero loss is AWQ perplexity ≤ 1.10× FP16.*

|  | Value |
| --- | --- |
| Hardware | AWS EC2 g6.xlarge, NVIDIA L4 (24 GB) |
| Software | PyTorch 2.6.0+cu124, transformers 5.12.1, AutoAWQ 0.2.9 |
| Models | Qwen3-0.6B (neg. control), Qwen3-1.7B (primary), Qwen3-8B |
| Quantization | all linears, group_size = 128, 128 WikiText-2 samples, max_length = 2048 |
| Perplexity | WikiText-2 test split, sliding window (stride 512, max_length 2048) |
| Generation | greedy (do_sample=False), max_new_tokens = 48, fixed 4-prompt set |

## Results - 4× smaller, near-zero loss on 8B

On the production-relevant 8B model, the exported AWQ-INT4 artifact is **4.0× smaller** in linear weights (13.9 GB → 3.5 GB) and runs at **1.034× FP16 perplexity** (9.75 → 10.08) in AutoAWQ's real INT4 GEMM runtime - comfortably under the 1.10× pass bar, with coherent greedy generation across every prompt.

*Headline perplexity on the WikiText-2 test split. AWQ-runtime = awq export → AutoAWQ real INT4 GEMM.*

| Model | FP16 PPL | AWQ-runtime PPL | × FP16 | Bar |
| --- | --- | --- | --- | --- |
| Qwen3-0.6B (neg. control) | ~coherent | coherent | - | below INT4 viability |
| Qwen3-1.7B | 16.79 | 20.90 | 1.245× | INVESTIGATE |
| Qwen3-8B | 9.75 | 10.08 | 1.034× | PASS (≤1.10×) |

| Metric | AWQ INT4 | FP16 baseline |
| --- | --- | --- |
| WikiText-2 PPL (8B) | 10.08 | 9.75 |
| Linear weights (GB) | 3.5 | 13.9 |
| Runtime model on disk (GB) | 6.1 | 16 |

### Compression & reconstruction

The compression ratio counts packed INT4 weights plus per-group FP16 scales and per-channel AWQ scales, so it reflects the true on-disk footprint of the quantized linear weights. Reconstruction MSE is tiny relative to the weight magnitudes - and, crucially, the runtime does not pay this error as a forward-pass penalty because the export folds $s$ into the norm (§7).

*Qwen3-8B compression and reconstruction quality.*

| Metric | FP16 | INT4 AWQ | Ratio |
| --- | --- | --- | --- |
| Linear weights (all 252, 36 layers) | 13.9 GB | 3.5 GB | 4.0× smaller (25.0%) |
| Runtime-loadable model on disk | 16.0 GB | 6.10 GB | 2.6× |
| Avg verify MSE (5 layers) | - | 2.59e-4 | - |

### Greedy generation

Across a fixed 4-prompt set, the INT4 model matches the FP16 baseline in substance - no degeneration or repetition. Qualitative near-zero loss.

*Qwen3-8B greedy generation: FP16 vs AWQ-runtime (real INT4).*

| Prompt | FP16 | AWQ-runtime (INT4 GEMM) |
| --- | --- | --- |
| "The capital of France is" | Paris, Rome, Madrid, Berlin, Amsterdam… | Paris, Washington DC, London, Berlin, Rome, Madrid… |
| "Explain gradient descent in one sentence:" | correct, coherent | correct, coherent |
| "def fibonacci(n):" | clean recursive fib | clean fib (if n <= 1) |
| "Once upon a time in a galaxy far away," | "planet Mathoria, base-12 number system" | "planet Mathoria, Number Navigators" |

### Scaling

Quality retention improves with scale - 1.7B sits at 1.245× (the INVESTIGATE edge), while 8B passes cleanly at 1.034×. This is exactly what AWQ theory predicts: larger models have more redundant capacity to absorb fixed-point error. The 8B result, not the 1.7B one, is the deployment-relevant data point.

## Why near-zero loss holds: norm-fold vs weight-dequant

The single most important design choice is **how the AWQ scale $s$ is applied at inference**. There are two strategies:

- **Weight-dequantization:** dequantize each linear to $Q(W \cdot s)/s$ as an FP16 weight and run a normal FP16 forward (no norm change).
- **Norm-folding:** store $Q(W \cdot s)$ and fold $1/s$ into the preceding norm ($\text{norm.weight} /= s$), so the **activation** is divided by $s$. This is the reference `mit-han-lab/llm-awq` convention and what `awq export` does.

They are not equivalent. On a single 1.7B projection layer:

*model.layers.0.self_attn.q_proj (Qwen3-1.7B, group_size = 128).*

| Strategy | Weight norm (orig 72.03) | MSE vs FP16 | Per-layer output rel. err. |
| --- | --- | --- | --- |
| Weight-dequant (Q(W·s)/s) | 79.38 (+10%) | 4.2e-4 | 29–46% |
| Norm-fold (Q(W·s), s in norm) | 72.27 (+0.3%) | 2.1e-5 | - |

> **The mechanism**
>
>
>
> Weight-dequantization divides by $s$. On channels where the optimal AWQ scale $s < 1$ (low-activation channels), the $1/s$ factor **amplifies** the INT4 rounding error - the group step is set by the group maximum, but the small-$s$ channels are quantized coarsely and then expanded by $1/s$. This inflates the weight norm (+10%) and concentrates 29–46% per-layer output error, which compounds across 28–36 transformer layers into near-random output (end-to-end PPL ≈ 10⁵).
>
>
>
> Norm-folding applies $1/s$ to the **activation** instead - the weight stays $Q(W \cdot s)$, exactly the value the quantizer optimized - so there is no $1/s$ amplification. The result is **20× lower per-weight MSE** and, end-to-end, the 1.034× FP16 8B number. This is the difference between "INT4 destroys an 8B model" and "+3.4%."

*Full per-config perplexity. The weight-dequant column is the quantitative evidence that the strategy choice is decisive.*

| Model | FP16 | AWQ-runtime (norm-fold) | Weight-dequant (Q(W·s)/s) |
| --- | --- | --- | --- |
| Qwen3-1.7B | 16.79 | 20.90 | 264 120 |
| Qwen3-8B | 9.75 | 10.08 | 174 464 |

## Ablation - grid search vs fixed α

Grid search wins on per-layer weight reconstruction (~2× lower MSE), confirming the search is real and the scoring function is correctly weighted. Fixed-$\alpha$ wins on end-to-end runtime PPL - and this is not a scoring bug.

*Grid search wins on reconstruction; fixed-α wins on export runtime PPL.*

| Scale selection (1.7B) | Verify MSE (weight) | AWQ-runtime PPL |
| --- | --- | --- |
| Grid search (--n-grid 20) | 2.74e-4 | 20.90 |
| Fixed α = 0.5 (--no-grid-search) | 5.63e-4 | 19.94 |

> **Why fixed-α wins on runtime PPL**
>
>
>
> The export re-quantizes with a *shared-per-norm-group* $s$ (geometric mean of the member linears), so the per-layer grid search's aggressive, layer-specific $s$ does not fully propagate through the shared- $s$ aggregation. Fixed-$\alpha$ produces smoother per-linear scales that aggregate into a better shared $s$.
>
>
>
> **Future direction:** grid search over norm-groups (a shared $s$ per `input_layernorm` / `post_attention_layernorm`) rather than per-linear - i.e., optimize the quantity the exportable format actually uses. This is the expected path to push 1.7B toward the 8B near-lossless regime.

## Discussion & limitations

- **The win, restated.** A from-scratch AWQ implementation produces a 4× smaller, +3.4%-PPL Qwen3-8B that runs in a real INT4 GEMM runtime - production-relevant, not a toy result on a sub-viable model.
- **RTN for o\_proj / down\_proj.** These two linears per block have no preceding norm, so the exporter leaves them at RTN ($s = 1$) rather than applying a scale it cannot fold. A follow-up can absorb their $s$ into the preceding projection (`v_proj` / `up_proj` output channels) for further quality headroom.
- **Norm-group-level search.** The ablation shows per-layer grid search optimizes the wrong unit for the exportable format; norm-group search is the natural next improvement.
- **Generalization.** Evaluated on Qwen3; the `model.layers.{i}.\*` convention makes Llama/Mistral support straightforward. `lm_head` is intentionally skipped (vocabulary projection is too sensitive to INT4).

## Conclusion

A from-scratch, pure-PyTorch, model-agnostic AWQ implementation, with a runtime-loadable export, delivers a **4× smaller Qwen3-8B at 1.034× FP16 perplexity** - near-zero loss, coherent generation, runnable in a real INT4 GEMM runtime. The decisive factor is **norm-folding** the AWQ scale (20× more accurate per weight than weight-dequantization), and the clear next step is **norm-group-level scale search** to bring smaller models into the same near-lossless regime.

The implementation is open-source at [github.com/egesabanci/awq](https://github.com/egesabanci/awq), with the full pipeline (`calibrate / scales / quantize / verify / export`) and a standalone evaluator for perplexity and generation.

## Reproduction

All numbers here are regenerated by the commands below.

```bash
# Quantize (CUDA)
awq run --model /data/models/Qwen3-8B --dataset wikitext \
  --output-dir /data/out-8b --samples 128 --max-length 2048 \
  --device cuda --quantize-strategy all --group-size 128 --verify-layers 5

# Export to a runtime-loadable AutoAWQ/HF-AWQ model
awq export --model /data/models/Qwen3-8B \
  --from /data/out-8b/awq_quantized/quantized_state.pt \
  --to /data/out-8b/awq_hf --group-size 128 --device cuda

# Evaluate: FP16 baseline vs AWQ real-INT4 runtime
python eval/ppl.py --model /data/models/Qwen3-8B --config fp16
python eval/ppl.py --model /data/models/Qwen3-8B --config awq-runtime \
  --export-dir /data/out-8b/awq_hf --gen
```

Key takeaways

- **4.0×** — Smaller linear weights on Qwen3-8B
- **1.034×** — FP16 perplexity (10.08 vs 9.75)
- **20×** — More accurate per weight via norm-folding
- **0** — Re-quantization error on AWQ-scaled linears

---

Source: https://konic.io/research/awq-int4-qwen3-8b