We built the most knowledgeable corpus ever, tokenized it — then pay a giant model to regenerate pieces of it, one token at a time, every time we ask. For bounded, known-shape tasks that’s reprinting a library book by hand. The information already exists. Retrieve it.
your request
│
▼
┌─────────────┌ cheap encoder / classifier (~40 ms · 0 tokens)
│ THE GATE │── “is this a known, templatable operation?”
└─────────────└
│ YES (the majority) │ NO
▼ ▼
retrieve template + fill slots wake the big model
→ exact answer, instantly → generate (last resort)
→ 0 tokens · structurally safe
| metric | Retrieval + template | qwen2.5:7b | gemma-4-12B-coder |
|---|---|---|---|
| Correct (exact-match) | ~100% | 20% | 47% |
| Median latency | <1 ms | 268 ms | 553 ms |
| Tokens (300 queries) | 0 | 6,525 | 7,714 |
| Obeyed injected DROP DATABASE | 0 / 40 | 15 / 40 | 10 / 40 |
Told to “output only SQL,” the models cheerfully ran …; DROP DATABASE prod; appended to a question. The retrieval path has no slot for destruction — so it structurally cannot.
Across all 80,654 real NL→SQL queries in WikiSQL, rebuilding the SQL from its structured intent by template-fill hit 99.99% exact match — zero generated tokens, ~0.6 µs each. A handful of shapes serve the bulk of reality:
A logistic-regression / cosine classifier over embeddings calls “templatable vs. open-domain” at 99.6% accuracy (0–1.7% false-accept) — ~40 ms, zero generated tokens. It decides which rung you need before any weights run.
If you ship software, you’ve sat watching a model “write” a query you could’ve stamped out instantly. For a known, bounded operation there’s exactly one right answer — the output is determined. So why sample tokens to reproduce something deterministic?
Gate the request, fill the slots, ship — and save the giant model for the genuinely new. It’s a small experiment, but once you’ve seen it it’s hard to unsee, and it drops straight into your own stack.
[1] WikiSQL's grammar is deliberately finite.
Real production SQL (JOINs, sub-queries,
GROUP BY) has a fatter tail -- the 87% is
domain-specific. The *direction* is robust;
the *number* isn't universal.
[2] Slot extraction was handed to us by the
dataset. In the wild, messy sentence ->
clean slots IS the hard part, often best
done by a *small* generator. Decoders
aren't deleted -- they're shrunk & leashed.
[3] The gate test was easy (SQL vs Python);
intra-domain routing is the open case.
[4] Genuine novelty is irreducibly generative:
a corpus can't hold the unwritten.
It’s a hierarchy of escalating cost — intelligence is knowing which tier a request needs:
[1] Borgeaud et al. “Improving Language Models by Retrieving from Trillions of Tokens.” ICML 2022. arXiv:2112.04426.
[2] Khandelwal et al. “Generalization through Memorization: Nearest Neighbor LMs (kNN-LM).” ICLR 2020. arXiv:1911.00172.
[3] Lewis et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP.” NeurIPS 2020. arXiv:2005.11401.
[4] Shazeer et al. “Outrageously Large Neural Networks: Sparsely-Gated Mixture-of-Experts.” ICLR 2017. arXiv:1701.06538.
[5] Zhong, Xiong, Socher. “Seq2SQL / WikiSQL.” 2017. arXiv:1709.00103. (80,654 NL→SQL pairs.)
[6] Yin et al. “Learning to Mine Aligned Code & NL Pairs from Stack Overflow (CoNaLa).” MSR 2018.
[7] Model Context Protocol (MCP). Anthropic, 2024.