Field guide · Operating substrate · June 2026
62% fewer tokens without losing the data that matters.
Every agent session pays a fixed tax before it does any work. The static context cascade, the stack of AGENTS.md files that tells an agent who the company is and how it runs, loads in full at the top of every session. On a substrate that has run for a year, that cascade pushes past 28,000 tokens before the first prompt, and the cost compounds across every parallel agent and every session in a day.
Posted June 26, 2026
The tax is invisible because it never appears as a line item. It surfaces later, as the agent that runs out of room sooner than it should: dropping earlier files from context, losing the thread on a job a leaner agent would have finished. On Rarefied Earth's own substrate, the primary workspace AGENTS.md alone reached 28,206 tokens before the first prompt.
The obvious fix is to trim the file, and the obvious fix is wrong. That AGENTS.md grew because the company kept adding the decisions, engagements, contacts, and pricing that agents actually read. The information is load-bearing. Cut it and the agents get dumber, a worse failure than the one being solved.
The real insight is that the file mixes two kinds of content with different compressibilities. Structural prose carries its meaning loosely; shorten it and a reader still navigates. Precision facts do not. A dollar amount, an email address, a ticket number, an engagement status: each is binary, and there is no lossy version of $48,250 that is still useful. That asymmetry is the design constraint: compress the prose hard, never run a precision fact through the compressor.
Two files from one source.
The distiller reads an AGENTS.md and writes two siblings next to it. The split is the architecture.
_context_facts.json is the safety layer: every precision fact in the source extracted verbatim and sorted into categories (dollar amounts, emails, phone numbers, dates, ticket numbers, percentages, URLs, version numbers, file paths, account identifiers), plus every source line that carries a fact, kept verbatim with its line number. It is never summarized or truncated, and the extractor errs toward over-capture, since a false positive only adds an already-correct line while a false negative loses a real fact.
_context_cache.md is the compressed view. Headings, code fences, rules, and table separators survive verbatim, so the skeleton stays navigable. Short lines pass through. Long prose lines get shortened, and a line carrying a precision fact is not cut on faith: it truncates to the lead-in and points to the facts file rather than risk leaving half a number behind.
In the source, a fact sits buried in a table row:
| Northwind | ACTIVE since 2025-03-12 ... retainer ~$48,250 ... |
In the facts file, the same values are pulled out and indexed, where nothing can shorten them:
{
"dollar_amounts": ["$12,400", "$48,250", "$9,750", "$185/hr"],
"ticket_numbers": ["ACME-142", "ACME-118", "PROJ-301"],
"email_addresses": ["dana@example.com"]
}
Load both and the agent has the full picture: structure from the cache, precision from the facts.
The gate that refuses to ship a lossy cache.
Before writing, the distiller verifies its own output. It round-trips the facts file and confirms every detected fact survives. If even one would be lost, it writes nothing and logs a red error rather than ship a cache quietly missing a number. It writes the facts file before the cache, so a crash between the two leaves both valid. The compressed view is an optimization; the facts layer is correctness-critical.
The numbers, on RE's own file.
Rarefied Earth's AGENTS.md, measured 2026-06-30:
- Reduction: 61.6 percent
- Original: 28,206 tokens
- Compressed cache: 10,822 tokens
- Preserved verbatim: 35 dollar amounts, 7 email addresses, 22 ticket numbers, 122 precision lines
Token figures are heuristic estimates, accurate to about plus or minus 15 percent against a cl100k tokenizer. The reduction percentage is the durable claim. The 61.6 percent is the structural compression of that one file, not total session cost, which depends on the full cascade. That file is usually the cascade's largest piece, so compression returns the most there.
Why a watcher, not a cron.
The cache is only useful if it is current. Consider when an AGENTS.md actually changes: a new engagement signs, a rate moves, a project flips from active to frozen, all high-stakes edits at unpredictable moments. A time-based cron is uncorrelated with them by construction, so it maximizes the window in which an agent reads a cache that still calls a closed deal open. Stale context does the most damage exactly when a clock is least likely to catch it.
So the trigger is the change itself. A launchd job runs every 5 minutes (StartInterval 300), hashes each watched file with SHA-256, and compares against the stored hash from the previous pass. When a hash moves, that one file is redistilled and nothing else is touched. The refresh is bound to the event that creates the risk, not to a clock that ignores it. It covers the iCloud-level, workspace, and per-project AGENTS.md, and adding another is one line in the watch list.
What the recovered budget buys.
An agent loading 10,822 tokens of cache instead of 28,206 of raw source starts with roughly 17,400 more tokens of working room. On a 200,000-token window that is the difference between holding three reference files during a refactor and being forced to drop one. The gain compounds the way the cost did: every parallel agent loads the cascade independently, so three agents recover the delta three times. The point is not the dollar saving, which is real but small. It is headroom: room for the work in front of the agent instead of for who the company is.
Where this stands.
One part of this is proven and the rest is day-1, stated plainly. The no-loss gate is the proven part. The distiller verifies its own output before writing, and on demand context_distiller.py --verify confirms the current cache against the current source with zero precision-fact loss. That check passes on RE's file as of 2026-06-30.
The rest is younger. Continuous freshness is bound to the change event by design, a property of the architecture, not a result proven across months of live edits. Per-session context cost is now instrumented: each Cursor session start appends one measured line to a running log, so the cost is captured live and beginning to accumulate, not yet measured at scale. Module 36 went into dogfood on 2026-06-26, running on Rarefied Earth as client zero, on a 30-day window that closes 2026-07-26 for accumulating exactly those metrics. It has not run for a paying client. We run it on ourselves first because that is the only test that separates infrastructure that works from infrastructure that demos.
This is Module 36 of Groundwork, the operating substrate Rarefied Earth builds for companies that run on AI and runs on itself before anything ships. If you operate a multi-agent system and want to know what the context layer does on your own stack, the address is joseph.scott@rarefied.earth.