LeastGen — 95% Fewer Tokens, 0ms Latency

Background

Autonomous AI coding agents have a dirty secret: most of their "reasoning" is actually expensive context repetition. Every turn, they re-send the project structure, the git status, the operating environment — information that was already available, just written in slightly different words. This is Agentic Context Inflation, and it's the dominant cost driver in agentic workflows.

LeastGen is the architecture that solves this. It sits between the agent and the inference provider as a transparent proxy, learns the structural patterns of agentic loops, and serves 95.1% of requests from local cache at 0ms latency — zero generated tokens, zero API cost. For the remaining 4.9% that genuinely need novel output, it routes to a pair of specialized local models pinned permanently in a single GPU.

The project is the operational implementation of the Principle of Least Generation: route first, generate last. Only reach for generation when nothing cheaper can answer the question.

System architecture

The system introduces a transparent, zero-configuration proxy layer into the agent stack:

⬡

LLM Gate Proxy

Listens on port 8766 and intercepts every agent-to-provider request. Instead of exact-match caching, it uses loose template induction — hashing the structure (roles and normalized instructions) of incoming prompts, not the literal text. Repetitive agent loops (git commands, directory checks) replay cached responses locally in 0ms.

◈

Multi-Turn Safety Gate

Detects multi-turn conversational loops and dynamically bypasses template caching, routing to live inference when the agent is in an active back-and-forth dialogue. Prevents loop hijacking while keeping the cache active for independent turns.

◎

1:1 VRAM-Pinned Models

Two specialized, pruned local models — a 3B Planner for step-by-step reasoning and tool selection, and a 4B Executor for syntax-perfect code generation — both locked in GPU VRAM concurrently via keep_alive: -1. Zero model-swapping latency on a single RTX A4500 (20 GB).

▦

Path-Aware Context Isolation

A shell-level and parser-level routing wrapper that maps the agent's memory to the developer's working directory. Converts paths into unique flat profile namespaces, ensuring different codebases never share state. Detects git repos and spins up isolated worktrees to prevent parallel agents from corrupting active files.

◇

Task Specialization Synergy

The Planner (3B) runs at lightspeed because it doesn't carry heavy coding weights — it just plans. The Executor (4B) wastes zero tokens on conversational filler — it outputs pure code. The proxy filters boilerplate in 0ms; when novel work is needed, the pinned models fire instantly.

◐

Concurrent HTTP Server

Built on a threaded Python HTTP server to handle concurrent agent and dashboard requests without socket timeouts or blocking. A live panel exposes the cache hit rate, savings, and recent request log for real-time monitoring.

Data flow

                            [Developer Workspace]
                                      │
                       (Auto-Profile & Git Worktree)
                                      ▼
                             [Hermes Agent CLI]
                                      │
                           (Standard OpenAI Client)
                                      ▼
                        [LeastGen LLM Gate Proxy] ──(95.1% Hit)──► [Local SSD Cache]
                         (llm_gate.py: Port 8766)                  (0ms / $0 cost)
                                      │
                                 (Cache Miss)
                                      ▼
                       [1:1 VRAM-Pinned Model Engine]
                        (RTX A4500 GPU: keep_alive = -1)
                        ├── Layer 1: 3B Planner (Thinking)
                        └── Layer 2: 4B Executor (Coding)

The key insight: not just any LLM works

Why not just use a single general-purpose model? Three reasons:

VRAM capacity. A single 32B or 70B model would either exceed the 20 GB of your typical workstation GPU or page-swap itself into unusability. By splitting the work across pruned, specialized 1:1 models (3B + 4B), both fit in VRAM concurrently at keep_alive: -1. There is no model-loading overhead — ever.

Task specialization compounds. The 3B Planner sheds all code-generation weights to focus purely on step-by-step logic and tool selection. The 4B Executor sheds all "thinking" patterns to focus purely on syntax-perfect code output. Neither wastes parameters — or tokens — on what the other does. This division of labour means each model punches well above its weight class.

The proxy is the multiplier. The cache handles the boring, repetitive structure of agentic workflows. When the agent needs to write novel code or make a decision, the pinned models respond immediately. The two optimizations multiply: one eliminates latency from repetition, the other eliminates latency from model loading.

Live production metrics

95.1% cache hit rate

715M tokens saved

$35.63 API costs avoided

2,772 requests processed

All figures recorded live on a single RTX A4500 workstation over 2,772 sequential agent executions. Of those, 2,637 were resolved from local cache — zero GPU, zero network, zero cost. The remaining 135 novel requests were handled by the VRAM-pinned 3B/4B pipeline at a total spend of just $1.45.

The cache hit rate holds across diverse workloads: SQL generation (84%), Claude Code agent traces (67%), and general Hermes agent loops (54%). The structured nature of agentic tool-use means the ceiling is high and the pattern generalizes — more template induction coverage pushes the rate higher with zero architectural changes.

Workspace governance

A practical problem LeastGen also solves: context contamination. Running multiple agent sessions from the same terminal means they share memory, SQLite databases, and state — leading to catastrophic cross-project pollution.

LeastGen's path-aware isolation automatically slugifies directory paths (e.g., ~/projects/app/src → projects-app-src) and creates unique flat namespaces for each. It detects git repositories and spins up sessions inside isolated git worktrees — so parallel agents working on different branches never touch the same files. The result: a single terminal agent becomes a decentralized fleet of isolated project managers, each with its own context, memory, and working tree.

Built with

Python Ollama / llama.cpp Hermes Agent OpenAI-compatible API Threaded HTTP Server Template Induction Hashing Git Worktrees RTX A4500 (20 GB) 3B Planner + 4B Executor

Why it matters

The AI industry has been optimizing for smarter models. LeastGen asks a different question: how much of what we're generating was already knowable? The answer — 95% — suggests that the next order-of-magnitude gain in agent efficiency won't come from a better LLM. It will come from not using an LLM at all for the boring bits.

Every agentic workflow has a fat tail of repetitive structure. LeastGen shows that a cheap proxy, a hash function, and two small models can clip that tail — on one GPU, with zero cloud dependency, for about the price of a coffee per thousand executions. The principle generalizes to any structured, tool-using agent: route first, generate last.

How much of your agent's work is cacheable?

If you run an autonomous coding agent — Hermes, Claude Code, Codex, or any OpenAI-compatible tool — you're paying for every turn. But not every turn deserves a model call. LeastGen is a single Python file, a proxy redirect, and two small Ollama models. You can deploy it in an afternoon and watch the cache hit rate climb.

What's your agent doing that it's already done before — and how much would you save by never generating it again?