Chapter 01 proved retrieval wins — once you already have clean slots. It said so plainly: the dataset handed us those slots, and turning a messy sentence into clean slots is the real work. That step is irreducibly a little generative. So we don’t delete the decoder — we shrink it and leash it: the smallest model that can fill the slots, wrapped in a grammar so it can emit nothing but valid slots. Rung 3 of the ladder, finally built.
messy sentence
│
▼
┌─────────────┌ small decoder (0.5–3B) — on a leash
│ THE LEASH │── grammar / JSON-schema constrained decoding
└─────────────└ it may emit ONLY tokens the schema allows
│
▼
validated slots {col, agg, where…} — well-formed by construction
│
▼
retrieval + template-fill (Chapter 01) → exact SQL
→ bounded tokens · no free-form output · injection has no slot
| metric | Retrieval + template (ch.01) | Leashed 0.5B | Leashed 3B | gemma-4-12B free (ch.01) |
|---|---|---|---|---|
| End-to-end correct | ~100% | — pending — | — pending — | 47% |
| Slot exact-match | given | — pending — | — pending — | — pending — |
| Median latency | <1 ms | — pending — | — pending — | 553 ms |
| Tokens (300 queries) | 0 | — pending — | — pending — | 7,714 |
| Obeyed injected DROP DATABASE | 0 / 40 | — pending — | — pending — | 10 / 40 |
Hypothesis: the leashed columns land near the retrieval anchor on correctness & safety, at a tiny fraction of the free-generation token cost. We’ll report what runs — not what we hope.
…; DROP DATABASE prod; appended). A schema-constrained decoder can emit only slot tokens — target 0 / 40, matching retrieval, beating free-gen’s 15/40 & 10/40./process/CHAPTER2_LEASHED_DECODER.md.Chapter 01 was almost too clean — of course retrieval wins when someone hands you the slots. The interesting question is the part I waved away: getting clean slots out of messy language.
My bet is you don’t need a big model for it — you need a small one that physically cannot color outside the lines. If that holds, Rung 3 is cheap and the whole ladder stands. If it doesn’t, that’s a real result too.
[1] Constrained decoding guarantees valid SHAPE,
not correct CONTENT. Valid-but-wrong is the
trap -- we score correctness, not just that
the JSON parsed.
[2] The grammar covers WikiSQL's slots. Richer
SQL needs a richer grammar -- ch.01's fatter
tail, now living in the schema.
[3] "Smallest model" is hardware- & prompt-
dependent. The accuracy-vs-size CURVE is the
finding; no single number is universal.
[4] If even 12B can't fill slots cheaply and
reliably, Rung 3 costs more than the ladder
implies -- a real, publishable negative.
The same ladder from Chapter 01 — this chapter builds the rung that was only drawn:
[1] Scholak, Schucher, Bahdanau. “PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models.” EMNLP 2021. arXiv:2109.05093.
[2] Willard & Louf. “Efficient Guided Generation for Large Language Models” (Outlines). 2023. arXiv:2307.09702.
[3] Geng et al. “Grammar-Constrained Decoding for Structured NLP Tasks.” EMNLP 2023. arXiv:2305.13971.
[4] Dong et al. “XGrammar: Flexible and Efficient Structured Generation for LLMs.” 2024. arXiv:2411.15100.
[5] Zhong, Xiong, Socher. “Seq2SQL / WikiSQL.” 2017. arXiv:1709.00103. (80,654 NL→SQL pairs.)
[6] Yu et al. “Spider: Large-Scale Cross-Domain Text-to-SQL.” EMNLP 2018. arXiv:1809.08887.