Edge intelligence, measured honestly.

Base

tiiuae/Falcon-H1-7B-Instruct

Architecture

Hybrid Mamba-2 SSD + attention, 7.59B params, 44 blocks

Compression

2.06 bits per weight average, mixed-precision PTQ

Size

3.48 GB, from 15.18 GB at F16

Runs on

stock llama.cpp, unmodified

Design philosophy

We prioritise factual retention over bit purity. Uniform ternary quantization left the model fluent and confidently wrong — it named Sydney as the capital of Australia. The Hessian explained why: ssm_out, the projection writing the Mamba-2 scan output, carried an outlier flatness of 6.78 against roughly 0.33 everywhere else. Holding it and ffn_down at Q6_K costs 0.23 GB and returns the model's world knowledge.

Quality

Two caveats we will not bury. The v2 run changed two variables at once — preserving ssm_out and raising calibration from 32 to 128 sequences — so the recovery cannot be attributed between them. And there is no same-size Q4_K_M baseline, so “better than conventional 4-bit at this footprint” is not a claim this data supports.

Performance

Running it

Requires a sampling chain — temperature 0.7, top_p 0.9, and repeat penalty 1.15 over a 2048-token window. The window, not the penalty strength, is the lever: at llama.cpp's default of 64 the sampler cannot see past the current turn, and cross-turn repetition measures 45% against 17% at 2048. It reduces looping rather than eliminating it.

On iOS the binding constraint is Metal's working set, not the jetsam limit — use n_ubatch = 256 on 6 GB devices. The open issues are listed in full on the model card.