FORGE
Mixed-precision quantization, chosen by the Hessian
Sub-3-bit quantization usually stops a model being a model. FORGE is a post-training pipeline that decides per tensor rather than uniformly: it measures Hessian flatness to find the weights that cannot survive ternary, holds those at higher precision, and crushes the rest. On Falcon-H1-7B that is a 4.4x reduction to 3.48 GB with factual recall intact.
- 2.06
- bits per weight, average
- 4.4x
- smaller than F16
- 12/12
- factual probes recovered
The failure that motivated it
Our first pass quantized every reachable tensor to ternary. The result was not a loss of fluency, which would have been easy to spot. It stayed articulate and became confidently wrong.
| probe | uniform ternary | mixed precision |
|---|---|---|
| carrot cake ingredients | "carrot cake mix, carrot cake mix, ..." | "Carrots, sugar, flour, eggs, butter, cinnamon..." |
| capital of Australia | "Sydney" | "Canberra" |
| boiling point of water | "99 C" | "100 C" |
What the Hessian showed
Across the reachable tensors, outlier flatness clustered around 0.33 — except for ssm_out, the projection that writes the Mamba-2 scan's output, which measured 6.78. A 20x outlier, and structurally explicable: it sits immediately after the scan, in the same position ffn_down occupies after the SwiGLU, where a fusable rotation cannot reach it. Crushing that tensor was destroying the path the model's factual recall travels through.
Excluding ssm_out and ffn_down from ternary and holding them at Q6_K costs 0.23 GB. Mean flatness across the still-quantized set drops from 1.1881 to 0.3818 — the outlier is simply no longer being crushed.
Measured
| build | size | ppl | vs F16 | factual probes |
|---|---|---|---|---|
| F16 | 15.18 GB | 6.5875 | 1.00x | 12/12 |
| uniform ternary | 3.25 GB | 10.7590 | 1.63x | 9/12 |
| mixed precision (shipping) | 3.48 GB | 9.1633 | 1.39x | 12/12 |
Honest limits
The shipping run changed two variables at once — preserving ssm_out and raising calibration from 32 to 128 sequences — so the recovery cannot be attributed between them. The flatness evidence makes ssm_out the likelier cause, but that is inference, not measurement. A third run would separate them.
- No same-size baseline. A standard Q4_K_M build has not been measured, so "better than conventional 4-bit at this footprint" is not a claim this data supports.
- Cross-turn repetition persists at roughly 17% over four turns even with a correctly sized penalty window. It is reduced, not eliminated.
- Meta-instructions fail. Direct instructions are followed; instructions about the conversation are not. That is a quantization ceiling, not a templating bug.
- Evaluated with targeted factual probes and perplexity, not MMLU or HellaSwag.