Why REMIT stops: the design decisions behind a decision system you can sign off
REMIT is built on four design decisions, none of them ours to invent: use a small model trained on one job, keep the facts in the documents, measure when the system is sure enough to act, and hand the rest to a person. This note explains each decision in plain terms and lists the published evidence behind it.
A regulated operator does not need a cleverer model. It needs a system it can put in front of an auditor. That constraint drove every design decision in REMIT, and the useful thing about those decisions is that each one rests on results other people have already published and measured. This note walks through the four, says what each buys a bank, an insurer or an agency, and ends with the references, numbered so you can check them.
1. One job per model, and a small model is enough
REMIT never asks a model to be a generalist. Each workflow gets its own small model, trained to choose from that workflow's fixed list of outcomes: refund the fee, ask for the reference, reject the claim, escalate. When the answer space collapses to a short list, the task stops being language generation and becomes classification, and classification is where small, fine-tuned models have repeatedly matched or beaten the largest general models on the same task [1][2][3]. In the most striking published case, a model over seven hundred times smaller than its comparator won on a three-way decision task [1].
What this buys the customer is not a benchmark score. It is that the model runs on a branch-class machine inside the perimeter, gives the same answer to the same input every time, and cannot produce an outcome that is not on the list. The last property matters most: a model that can only pick from your list cannot invent a remedy your policy does not contain.
The limit, stated: most of the published comparisons are against general models used without task-specific training, on classification-shaped tasks. That is precisely the shape REMIT restricts SIFT to, which is why SIFT is never asked to summarise, draft or reason in free text.
2. Facts live in the documents, not in the model
Small models cannot memorise a policy manual, and quantised small models memorise even less: the measured capacity is roughly two bits of knowledge per parameter, and it degrades further at the precision levels that make on-premise deployment cheap [4]. REMIT treats this as a design rule rather than a limitation. FOLIO reads every case from its source documents and the local policy corpus, and every field in the case record is traced to the line it came from. The model decides; the documents know.
For the customer this is the difference between a system that says "refund" and a system that says "refund, because Fee Schedule 4.2 allows reversal of a duplicate draw within 30 days, and here is the line on the statement". The second one is a reason. A reason is what a complaint response, a claims letter and an audit file are made of.
3. Measure when the system is sure enough, on your own cases
The heart of REMIT is not a model at all. Before go-live, the system is scored on hundreds of the customer's own past decisions that it has never seen. From that measurement it derives a margin: how far ahead the top outcome must be before the system is allowed to act on it. The procedure is called conformal prediction, and its property is the one an auditor cares about: given that measurement set, the true outcome will be among the outcomes the system keeps open at least as often as the standard the customer set, for any model and any data [5][6][7]. Not a vendor estimate. A theorem.
This is what BOUNDS does with every case. If exactly one outcome survives the margin, the case is handled and sealed. If two survive, the machine may not act; a person decides between them with the evidence already assembled. The number the head of compliance reads on the oversight screen, right 91 percent of the time across 512 similar past cases against a required 90, is that measurement, stated in words.
The limit, stated: the guarantee holds while live cases resemble the cases it was measured on. When they stop resembling them, the guarantee is void. REMIT therefore monitors for drift, re-measures on a schedule, and treats a stale measurement as a reason to send every case to a person until it is refreshed. Coverage is a frequency across cases, not a promise about any single one, and it does not hold against someone deliberately shaping the input stream.
4. Hand off, do not guess
Systems that let a small model handle what it is confident about and pass the rest to something larger are well studied. The published results show large reductions in cost and latency [10][11], and one failure mode that shows up whenever the small model's confidence is set by hand or does not actually track its accuracy [8][9][10]. REMIT's answer to that failure is decision three: the hand-off threshold is measured, not set, and it is measured on the customer's own cases. The hand-off itself is a product surface, not an error path. BENCH shows the reviewer the evidence first, a suggestion clearly marked as a suggestion, and the same list of outcomes; the reviewer decides, and the record says so.
Every hand-off a person resolves becomes a labelled example. That is the loop that makes REMIT improve with use: the next measurement round includes the cases people decided, and the margin tightens where the system has earned it. A generative deployment has no such loop; ours is a consequence of the architecture.
What we chose not to do
There are published ways to train a model to report its confidence more accurately, by rewarding calibration directly during training [12][13]. We keep these in reserve. They cost accuracy and training stability, and they produce a confidence number rather than a guarantee. Measuring on held-out cases is training-free, gives the guarantee, and leaves the model alone. If a workflow ever needs sharper per-case confidence on top of the guarantee, that is where we would look next.
What none of this proves
That any particular solution transfers cleanly to a particular institution's data. The public banking corpus we build the first solution on proves the mechanism; accuracy on your complaints is measured on your complaints, before anything goes live, and the number is on the record. We do not conflate the two.
References
[1] Distilling Step-by-Step
arXiv:2305.02301Hsieh et al., ACL 2023 Findings
A fine-tuned 770M model outperforms a 540B model used few-shot on a three-way decision task; a 220M model wins on another.
[2] Fine-tuned small encoders versus zero-shot LLMs on text classification
arXiv:2406.08660Bucher & Martini, 2024
Fine-tuned DeBERTa-v3, RoBERTa and ELECTRA consistently outperform zero-shot GPT-3.5, GPT-4 and Claude on text classification.
[3] Fine-tuned DeBERTa versus GPT-4o on eight-way emotion classification
arXiv:2505.000132025
Fine-tuned DeBERTa-base (0.870) and DeBERTa-large (0.860) outperform GPT-4o (0.844).
[4] Physics of Language Models, Part 3.3: Knowledge Capacity Scaling Laws
arXiv:2404.05405Allen-Zhu & Li, 2024
Roughly 2 bits of knowledge per parameter, holding at 8-bit and degrading at 4-bit precision.
[5] A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
arXiv:2107.07511Angelopoulos & Bates, 2021
The split-conformal construction and its coverage theorem, for any model and any distribution, under exchangeability.
[6] ConU: Conformal Uncertainty in Large Language Models
arXiv:2407.004992024
Conformal uncertainty for language-model outputs with correctness coverage guarantees.
[7] Prune 'n Predict
arXiv:2501.005552025
Conformal prediction sets over multiple-choice outputs; the set-size signal REMIT uses to decide whether the machine may act.
[8] GATEKEEPER
arXiv:2502.193352025
Fine-tunes a small model's confidence specifically to improve hand-off decisions.
[9] Calibration-aware cascades
arXiv:2609.114462026
Gives hand-off thresholds an explicit selective-risk interpretation.
[10] Confidence-based cascade scoring
arXiv:2604.197812026
76% cost and 61% median-latency reduction versus always using the large model, escalating 7% of items; fails when the small model's confidence is degenerate.
[11] Training-free semantic-agreement cascades
arXiv:2509.21837CMU, EMNLP 2025
Matches or surpasses target-model quality at 40% of the cost across models from 500M to 70B parameters.
[12] Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
arXiv:2507.16806Damani et al., 2025
Rewarding calibration during training (Brier score) yields models that are both accurate and calibrated.
[13] Rewarding Doubt
arXiv:2503.02623Stangel et al., ICLR 2026
Uses the logarithmic scoring rule as a reinforcement-learning reward for calibrated confidence.