This series began with a quiet question: how much of what a large language model generates was already knowable? Chapter 1 proved the principle on a bounded task — WikiSQL — where a cheap gate plus deterministic retrieval beat a 12B model on cost, speed, and safety with zero generated tokens. Chapter 2 set out to build Rung 3 of the ladder: a shrunk, grammar-leashed decoder that could turn messy language into clean slots cheaply enough to make the hierarchy practical.
The answer — 95.1% — is now running in production on a single RTX A4500, serving real agent traffic every day. The project is called LeastGen, and it is the operational realization of every idea this series explored.
LeastGen is a transparent inference optimizer that sits between any OpenAI-compatible agent (Hermes, Claude Code, Codex) and the inference provider. It intercepts every request, hashes its structure — not the literal text — and serves 95.1% of them from local SSD cache at 0ms latency, zero GPU, zero cost. The remaining 4.9% that genuinely require novel output are routed to two specialized local models pinned permanently in VRAM: a 3B Planner for step-by-step reasoning and a 4B Executor for syntax-perfect code generation. Both fit on one GPU because they are pruned and specialized — no parameter is wasted on what the other does.
[Developer Workspace]
│
(Auto-Profile & Git Worktree)
▼
[Hermes Agent CLI]
│
(Standard OpenAI Client)
▼
[LeastGen LLM Gate Proxy] ──(95.1% Hit)──► [Local SSD Cache]
(llm_gate.py: Port 8766) (0ms / $0 cost)
│
(Cache Miss)
▼
[1:1 VRAM-Pinned Model Engine]
(RTX A4500 GPU: keep_alive = -1)
├── Layer 1: 3B Planner (Thinking)
└── Layer 2: 4B Executor (Coding)
The architecture validates the cost ladder from Chapter 1 empirically. The cheap gate (Chapter 1's Rung 1) is the template-induction proxy. The retrieval path (Rung 2) is the SSD cache. The shrunk, leashed decoder (Rung 3, the subject of Chapter 2) is the 3B Planner + 4B Executor pair — small models that are never trusted with free-form generation, only bounded slot-filling within their specialization. And the big model (Rung 4) is the fallback for genuinely novel requests that clear the cache.
Every claim in this series — that the query space is Zipfian, that a gate can route without generating, that context inflation is the dominant cost — is now backed by live production data across 2,772 agent executions. The 84% template coverage on SQL, the 67% on Claude Code traces, the 54% on general Hermes agent loops — each number is a measured floor, not a simulated ceiling.
The principle — route first, generate last — generalizes to any structured, tool-using agent pipeline. LeastGen is one implementation of that principle, but the architecture is not specific to its components. Any cheap gate, any deterministic retrieval layer, any pair of small pinned models. The multiplier is the idea: most of what we generate is already knowable.