The Edge AI Physics of FORGE and HELIX: Forcing a 7B Hybrid onto an iPhone

We set out to answer a specific systems question: can you run a 7B-class assistant locally on an iPhone, without custom hardware, and without the OS killing it? We built two pieces to find out. FORGE is a post-training quantization pipeline; HELIX is a Metal prefill kernel for Mamba-2 hybrids that plugs into llama.cpp. Both work. The stack boots and answers questions on an iPhone 14 Pro Max.

The more useful result is that we were wrong about the constraint. We spent the project designing around a memory limit that never bound, and the thing that actually stopped us took about ten minutes to find once we measured on hardware instead of reasoning from a spec sheet.

Compression: what 2.06 bits costs

An FP16 Falcon-H1-7B is 15.18 GB. To fit a phone we needed roughly a quarter of that, which means sub-3-bit territory, which is where models usually stop being models. Our first pass quantized everything to ternary and the result was worse than a loss of fluency: it stayed articulate and became confidently wrong.

The fix came from the Hessian. Across the reachable tensors, outlier flatness sat around 0.33 -- except for ssm_out, the projection that writes the Mamba-2 scan's output, which measured 6.78. A 20x outlier. Crushing that tensor to ternary was destroying the path the model's factual recall travels through. Excluding ssm_out and ffn_down from ternary and holding them at Q6_K cost 0.23 GB and bought the knowledge back.

Two caveats we are not going to bury. The v2 run changed two variables at once -- preserving ssm_out and raising calibration from 32 to 128 sequences -- so the recovery cannot be attributed between them. The flatness evidence makes ssm_out the likelier cause, but that is inference, not measurement. And we have no same-size Q4_K_M baseline, so "better than conventional 4-bit at this footprint" is not a claim this data supports.

Acceleration, and Amdahl's tax

HELIX reassociates the SSM recurrence into a chunked associative scan, so the work lands on matrix hardware instead of a serial loop, and keeps the carried state in threadgroup memory instead of spilling it to device memory every chunk. Against upstream llama.cpp's own Metal SSD kernel it is a real win. Against the wall clock it is a much smaller one.

Falcon-H1 is a hybrid: the SSM scan is a minority of the computation. Ternary dequantization across the rest of the network dominates. A 2.57x speedup on a minority component is a 1.36x speedup overall, and no amount of further kernel work changes that ratio. If you are optimizing an edge model, measure which component owns the time before you spend a month on one.

The iOS reality check

We cross-compiled for arm64-apple-ios, signed with the increased-memory-limit and extended-virtual-addressing entitlements, and pushed the 3.48 GB model onto a device. It loaded in 6 to 8 seconds and answered. Then we looked at the numbers we had instrumented, and the premise of the entire memory strategy fell over.

[mem] at failure: footprint 0.30 GB, available 3.70 GB, limit ~4.00 GB, device RAM 5.50 GB
error: Insufficient Memory (kIOGPUCommandBufferCallbackErrorOutOfMemory)
llama_decode: failed to decode, ret = -3

The process footprint was 0.30 GB. Not 3.7. iOS jetsam compares phys_footprint, and mmap'd model weights are clean, file-backed pages that are not charged to it -- so the limit we had spent the project engineering around was never remotely approached. The entitlement worked exactly as intended and was irrelevant, because it raises the jetsam ceiling and the ceiling that actually binds is Metal's recommendedMaxWorkingSetSize.

Dropping n_ubatch from 512 to 256 halved the compute buffer and put us at 4018 of 4096 MiB. That is the whole fix. It is also 98% occupancy, which is why prefill completes but is occasionally SIGKILLed -- so no, we did not entirely avoid the memory kill.

There is 312 MiB of pure waste in that table. The mapped model is 3313.51 MiB, but because it exceeds Metal's maxBufferLength, ggml splits it into two overlapping buffer views and Metal charges the working set for both. Reclaiming the overlap is worth more than every other memory lever we have.

What it feels like to use

A 146-token reply takes 26 seconds. That is 18 GB/s of effective bandwidth against the weights, roughly half the A16's LPDDR5 ceiling, so decode is bandwidth-bound rather than broken -- and HELIX accelerates prefill only. On a phone, the user judges the model on decode. We optimized the half that was already fast.

The verdict

FORGE and HELIX both did what they were built to do. FORGE demonstrates that mixed-precision PTQ guided by Hessian flatness can hold factual recall at 2.06 bpw where uniform ternary destroys it, and the ssm_out finding should generalize to other Mamba-2 hybrids. HELIX is a legitimate upstream contribution to llama.cpp: it beats the existing Metal SSD kernel on its own supported shape and covers three shapes that kernel declines outright.

For consumer iOS on current hardware, a dense 7B is the wrong shape. Not because it does not fit -- it does -- but because at 5.5 tok/s the thing that limits the product is memory bandwidth, and no kernel we write changes how fast an A16 can stream 3.3 GB of weights. The honest paths forward are a 3B payload, or the 12 GB hardware in the iPhone 17 Pro, or a fused decode kernel. Probably all three.

One more thing we got wrong, for the record: the HELIX MPP path -- the one benchmarked at 4.62x on the M5's neural accelerators -- has never engaged for this model on any hardware. It is compiled for d_state = 128 and Falcon-H1 uses 256, so the runtime silently falls back. We only found that by logging which backend actually resolved, rather than trusting the flag we had set.

Open issues

• —No same-size Q4_K_M baseline, so the value of mixed-precision at this footprint is unproven.
• —312 MiB of overlapping buffer views inflate the GPU working set and keep the app at 98% of its budget.
• —Decode needs a fused single-step kernel; prefill acceleration does not reach the thing users feel.
• —HELIX_MPP_K is hardcoded to 128 and needs dynamic extents to reach d_state = 256 on M5 and A19 Pro.

All weights, scripts, and source are public under the SG Systems GitHub and Hugging Face organizations.