Your agentic app just ran a search. The tool returned 500 results as JSON. Your agent appended all of it and fired off an API call — 45,000 tokens to answer a question that needed maybe 4,500.

Tejas Manohar, a senior engineer at Netflix, hit this problem every day. He was running out of tokens in Claude Code sessions and couldn’t figure out where they were going. When he finally dug in, he found tool calls were pulling entire log files into the context window, and 90% of it was noise. A CPU spike crashed his laptop once; Claude had read the full log verbatim to help investigate it.

That frustration became Headroom, an open source project now at 18.7k GitHub stars with 30+ contributors, four months after going public. The project reports 200 billion tokens saved across its user base (opt-in telemetry, so the real number is higher), which translates to roughly $700,000 in avoided API costs.

Headroom sits between your app and the LLM and compresses what the agent reads — tool outputs, logs, RAG results, API responses — before those tokens reach the model. Typical savings in real coding sessions: 20–30%. For high-redundancy content like large JSON arrays or build logs, the compressors reach 83–95%. There’s a CCR layer to make compression recoverable when the model needs something that was compressed away.

Three stages in the architecture. Provider cache mechanics that are trickier than the docs let on. Failure modes worth knowing before you deploy.

Table of contents

• The problem
• The existing landscape
• Architecture: three-stage pipeline
  • Stage 1: CacheAligner
  • Stage 2: ContentRouter
  • Stage 3: IntelligentContext
• CCR: reversible compression
  • How it works
  • BM25 search within compressed data
  • Message-level CCR
• Cache savings: how they compound
• Memory: cross-agent persistence
• Use cases beyond coding agents
• Integration
  • Proxy mode
  • SDK
  • Framework wrappers
  • MCP
• Benchmarks
• Trade-offs
• What’s next: Headlight
• Conclusion

The problem

Context in agentic apps is structurally different from single-turn chat. In a chatbot, context is bounded and predictable. In an agent loop:

• Tool calls return arbitrarily large payloads
• Conversations accumulate over many turns
• RAG pipelines inject retrieved chunks, many irrelevant to the current query

Truncation is the naive fix, but it discards the tail, not the least important content. A fatal error at position 67 in a 100-line log gets cut before you see it.

The subtler fix — prompt engineering, chunking, summarization — requires manual work per use case and still forces a choice between completeness and cost.

Most token compression research has focused on user prompt compression: condensing verbose natural language inputs into semantically equivalent shorter ones. But in a coding workflow, the user prompt is rarely the bottleneck. File reads, web pages, archive papers, database payloads, CLI outputs — these are the actual source of bloat. And they don’t compress the same way.

Headroom’s bet: content has structure, and most of what consumes tokens isn’t the structure — it’s the noise. Compress the noise by type, preserve the structure, let the model retrieve full data when it needs it.

The existing landscape

Several tools address parts of this problem, and Headroom was designed with awareness of them.

Provider-native compaction. Claude and OpenAI summarize context when you approach window limits. It’s extremely lossy: the entire accumulated context becomes a flat markdown file, and you lose the structure between messages, tool calls, and results.

KV cache. Every message you send to an LLM isn’t just your new message; it’s the entire conversation history appended. Providers cache the attention states for prefixes they’ve seen before. If your prefix hasn’t changed, you pay 10–50% of normal input cost. If anything in that prefix changed (even a session UUID in your system prompt), it’s a full-price miss. Providers expose this but don’t always make the conditions transparent.

RTK (Token Killer). Wraps CLI tool calls that coding agents make (e.g., gh commands for GitHub) and automatically adds compression flags like --compact to reduce verbose output before it reaches the context window. Targeted and effective for CLI-heavy workflows.

LeanCTX. A similar approach to per-call output compression. Another CLI-level interceptor.

Commercial options. Companies like Compressor and TokenCompany (both YC-backed) expose cloud compression APIs. Send a payload, get a compressed payload back. The problem: each provider exposes different APIs, different cache semantics, and different edge cases. Building reliable integrations across all of them is genuinely hard to get right.

LLM Lingua. Microsoft’s open source project for text compression via token importance scoring. Trained on meeting summaries. The model works, but meeting summaries are a poor proxy for coding agent traces, and the domain mismatch hurts accuracy.

Headroom’s position: local proxy, pip install, no per-provider integration work, different compressors for different content types, and compression that’s recoverable when the model needs what was compressed away.

Architecture: three-stage pipeline

Every request flows through three stages:

CacheAligner → ContentRouter → IntelligentContext

Headroom integrates via 11 hooks that Claude Code and OpenAI expose — pre-session, pre-tool-call, post-tool-call, and others. These hooks are where Headroom intercepts, compresses, and injects retrieval tooling.

Stage 1: CacheAligner

CacheAligner stabilizes the message prefix to get provider-level KV cache hits.

LLM providers cache computed attention states for prompt prefixes they’ve seen before. A cache hit cuts input token costs 50–90%. Dynamic content in system prompts causes cache misses on every request:

Before:
"You are helpful. Current Date: 2025-04-06"  ← changes daily, no cache hit

After:
"You are helpful."                            ← stable prefix, cache hit
"[Context: Current Date: 2025-04-06]"         ← dynamic part moved to tail

CacheAligner extracts date strings, session IDs, and other dynamic content from system prompts and appends them instead of injecting them at the start.

What Claude’s docs don’t tell you. Anthropic’s default prefix cache TTL is 5 minutes. Continuous interaction within that window gets the 90% read discount. At minute six, you pay for the whole array again. The catch: if Claude forks a sub-agent to handle part of your task, the sub-agent has its own 5-minute prefix cache. By the time it returns, your TTL has elapsed. There’s also a 1-hour TTL option, which costs 2× for writes but gives 90% savings on reads, better for users who come back to resume long sessions. For rapid back-and-forth, the 5-minute default wins. Headroom has a feature in progress that reads your historical session patterns and picks between the two automatically.

Stage 2: ContentRouter

ContentRouter auto-detects content type and routes each message to the right compressor.

Detection uses ML classification combined with pattern matching. What each compressor preserves:

• SmartCrusher (JSON): keys, brackets, booleans, nulls, short values, UUIDs. Squashes long strings and whitespace. Uses the user’s query to determine which fields matter, then calculates mean and standard deviation across field values to identify and drop outliers. When the LLM retrieves data because it needed something that was compressed, the system records that and compresses less aggressively next time.
• CodeCompressor: imports, function signatures, class definitions, types. Compresses function bodies and comments. Uses AST parsing rather than line-based heuristics, so it understands the structure of the code rather than treating it as text.
• LogCompressor: timestamps, log levels, error messages, stack traces. Drops repeated patterns.

These compressors are structure-preserving, not truncating. A log compressor that knows what matters in a log file beats one that keeps the first N lines.

CompressBase: the fallback text compressor. For content that doesn’t fit a structured compressor, Headroom uses CompressBase, an encoder-only model trained on agentic traces. It looks at a token and asks whether its presence or absence changes the output. It doesn’t generate or summarize — it filters. Encoder-only means it’s fast enough to run locally without a GPU.

This was built because LLM Lingua, the available prior art, was trained on meeting transcripts. Meeting transcripts and agentic tool call traces have very different characteristics, and a model trained on one doesn’t transfer well to the other. CompressBase is open source. By the team’s own account, it’s first-generation: better training data from more diverse agent traces is the obvious next step.

Stage 3: IntelligentContext

After per-message compression, IntelligentContext scores the full conversation and drops the lowest-value messages to fit the token budget.

Six weighted factors:

• Recency (0.20): exponential decay from the end of conversation
• Semantic similarity (0.20): cosine similarity to recent context
• TOIN importance (0.25): learned retrieval rate — messages users frequently retrieve via CCR score higher going forward
• Error indicator (0.15): ML-based error detection via field semantics, not keyword matching
• Forward reference (0.15): messages that later messages depend on are preserved
• Token density (0.05): dense messages score higher than repetitive ones

The system prompt is never dropped. The last N turns are always preserved. Dropped messages go into the CCR store.

CCR: reversible compression

Aggressive compression has an obvious problem: you might throw away something the model needs.

CCR (Compress-Cache-Retrieve) handles this. The project calls it “reversible compression,” which is a loose use of the term, but the idea is sound. Compress aggressively, cache originals locally, give the LLM a retrieval tool to fetch full data on demand. If the compressed summary isn’t enough, the model can ask for more.

CCR is backed by local Redis and SQLite with a 5-minute TTL by default. Data older than that requires explicit TTL configuration. Storage is the constraint: keeping originals around indefinitely isn’t practical on a local machine.

How it works

When SmartCrusher compresses a 1,000-item JSON array down to representative items, it:

1. Stores the original in the CCR store keyed by a hash
2. Adds a marker: [1000 items compressed. Retrieve more: hash=abc123]
3. Injects a headroom_retrieve tool into the LLM’s available tools

If the compressed data is enough to answer, the model answers. If not, it calls headroom_retrieve(hash="abc123"). The response handler fetches from local storage and injects it into the conversation. The client app never sees this.

In practice: in testing against GPT-4o Mini, retrieval calls work reliably. In real sessions, users see the LLM call headroom_retrieve less than 1% of the time; it finds answers from the compressed summaries. When it does call, it’s working as intended.

The MCP tool declaration tells the model when to call it: “If you’re unable to find information or something appears compressed, call this tool with the provided hash.” The LLM is smart enough to recognize the marker and make the call.

BM25 search within compressed data

The LLM can search within cached data rather than retrieving everything:

{
  "name": "headroom_retrieve",
  "parameters": {
    "hash": "abc123",
    "query": "authentication errors"
  }
}

This runs BM25 over cached items and returns only the relevant subset.

Message-level CCR

When IntelligentContext drops messages to fit the budget, those go into CCR too:

[Earlier context compressed: 60 message(s) dropped by importance scoring.
Full content available via ccr_retrieve tool with reference 'def456'.]

The model can recover earlier turns if the current query depends on them.

There’s a feedback loop here: when the model retrieves dropped messages via CCR, the system records the pattern and scores similar messages higher in future sessions. Drop accuracy improves over time.

Cache savings: how they compound

CacheAligner and provider KV caching work independently, but they stack. A 100K token input through the full Anthropic pipeline:

1. SmartCrusher compresses to 20K tokens (80% reduction)
2. CacheAligner stabilizes the prefix; 18K of those 20K hit the cache
3. Anthropic’s cache read discount is 90% off input price

Effective cost: 2K full-price tokens + 18K at 10% = 3.8K token-equivalents. That’s 96.2% reduction from the 100K baseline.

Headroom handles the cache_control tagging automatically for Anthropic. Google’s implementation is described by the project as unreliable in practice. OpenAI’s is automatic but less aggressive on savings.

These discounts only activate when prefixes are stable, which is what CacheAligner is designed to ensure.

Memory: cross-agent persistence

Headroom includes a memory feature that uses a local SQLite-backed knowledge graph to persist information across agent sessions and across different agents.

One agent records facts (“I prefer dark mode,” “we’re using Go 1.22 for this project”), another agent reads them. The storage is local. The sync is bidirectional between Claude Code and Codex sessions, so context from one coding environment carries forward to another.

This is less novel since Claude Code introduced MEMORY.md natively. The cross-agent part is still useful, though: a managed variant could aggregate learnings across multiple users, seeding one developer’s agent with institutional knowledge from another’s sessions. The local version is a good building block if you’re running several agents in sequence on related tasks.

Use cases beyond coding agents

The project was built for coding agents, but several production uses have emerged in other domains.

Voice agents. Human-perceptible latency is roughly 200 milliseconds. Most voice agent pipelines run: speech-to-text → LLM call → text-to-speech. The LLM call, when it involves tool use and retrieval, often pushes total latency above 300ms. One team is using Headroom specifically to compress the data flowing into that LLM call, cutting latency under the perceptible threshold. When the bottleneck is latency rather than cost, the token reduction directly translates to response time.

Image and video. A company working in industrial settings sends video recordings of machine operators to Claude for analysis; the output is step-by-step operating instructions. Cost before Headroom: $3 per video upload. Cost after: $0.20. The approach chops video into frames, compresses token usage across the image data, and sends only what’s necessary. A 15× cost reduction on a repeated workflow.

The compressors work beyond coding workflows. Domain-specific variants for financial and medical data are on the roadmap. Those documents don’t compress like code or logs — numbers and legal clauses need different handling than variable names and stack traces — and the project is aware of that.

Integration

Headroom supports four deployment models.

Proxy mode

Run the proxy, point your existing client at it. The dashboard at localhost:8787 shows real-time savings, cache prefix hits, and compressor usage.

pip install "headroom-ai[proxy]"
headroom wrap claude    # or: headroom wrap codex

# Or manually:
headroom proxy --port 8787
ANTHROPIC_BASE_URL=http://localhost:8787 claude

Works for any client that supports a base URL override.

SDK

from headroom.compression import compress

result = compress(messages, model="gpt-4o")
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=result.messages
)
print(f"Saved {result.tokens_saved} tokens ({result.compression_ratio:.0%})")

The TypeScript SDK requires the proxy running separately (compression is Python-only).

Framework wrappers

Drop-in wrappers for LangChain, Agno, and the Vercel AI SDK:

# LangChain
from headroom.integrations.langchain import HeadroomChatModel
llm = HeadroomChatModel(ChatOpenAI())

# Vercel AI SDK
import { withHeadroom } from 'headroom-ai/vercel-ai'
const model = withHeadroom(openai('gpt-4o'))

MCP

Three tools for Claude Code or Cursor: headroom_compress, headroom_retrieve, headroom_stats.

headroom mcp install && claude

Benchmarks

From the documentation and community-reported data (not independently verified):

These are best-case scenarios, weighted toward high-redundancy inputs (large JSON arrays, bulk logs). For typical mixed coding sessions, the project reports 20–30% savings, a function of how much of the token budget comes from tool call outputs versus user prompts or system context.

The 200 billion token figure comes from opt-in telemetry that reports only the number of tokens saved, no content. Compressor usage statistics are stored locally and queryable via the dashboard. Independent benchmarks on your specific tool outputs will be more predictive than these numbers.

Accuracy benchmarks are ongoing. The evals compare task completion — whether the agent arrived at the same answer before and after compression — across standardized coding benchmarks. CCR’s reversibility means accuracy should match the uncompressed baseline, and in testing so far it does. The project runs weekly drift detection jobs and plans per-version evaluations as models update.

Trade-offs

Compression works well on high-volume, homogeneous data: search results, log arrays, database rows, JSON API responses. These have structural redundancy the compressors can exploit.

It can hurt on tasks requiring exact fidelity — legal document review, code audits where subtle bugs matter, anything where “similar” and “identical” aren’t interchangeable. CCR shifts this from a hard limit to a soft one, but retrieval depends on the model recognizing that it needs more, which requires the compressed summary to give enough signal.

PII, PHI, and identity data. Headroom has an in-progress PR to strip PII, PHI, and identity fields before compression begins. CompressBase is trained to avoid compressing UUIDs, links, and identifiers. If something does get compressed that shouldn’t have been, the CCR layer can retrieve it. For high-compliance environments, verify what the current handling actually does before relying on it.

Token source assumption. Headroom assumes tool outputs are the primary source of token bloat. If your tokens are dominated by a long system prompt or long user query, you’ll see smaller gains.

Proxy dependency. The TypeScript SDK requires a locally running Python proxy. You need to manage its lifecycle, monitor it, and handle its failures — a failure mode that doesn’t exist in a direct LLM call.

CCR TTL. Originals are cached for 5 minutes by default. headroom_retrieve calls for older data will fail silently: the model won’t get what it asked for, and it won’t know why. Longer sessions need explicit TTL configuration.

Multi-process deployments. CCR uses local Redis and SQLite. Multiple proxy instances mean cache misses on headroom_retrieve calls. The architecture supports pluggable storage backends, so replacing local SQLite with RDS on AWS is possible, but you handle that configuration yourself. The docs don’t cover it.

Enterprise proxy integration. Many enterprise teams already route Claude through an internal LLM proxy (LiteLLM is common). Adding Headroom creates a proxy-behind-proxy setup. Integration work to make it cooperate with LiteLLM is in progress; for now, enterprise deployments may need custom plumbing.

Latency. Headroom adds ~1–10ms per content type before the LLM call. Not zero. Worth measuring in latency-sensitive pipelines.

Version drift. CompressBase is in active development, TOIN-based scoring learns from usage, and provider API behavior changes. Retest accuracy when upgrading.

What’s next: Headlight

Headroom sees every token flowing between agents and models. The project is using that position for something beyond compression: context provenance.

Provenance means being able to confirm what went into a context window and where it came from. Foundation models don’t track this. They don’t know or care where your data originated, and the information doesn’t survive across providers. Organizations running multi-agent pipelines that span multiple model providers have no provider-agnostic way to track source metadata.

They’re building a project called Headlight to address this. Most observability tooling today — OpenTelemetry, OpenLLMetry — produces dashboards designed for humans to read. But agents are increasingly the ones consuming telemetry data, debugging their own runs and auditing their own tool use. Telemetry output needs to be token-efficient for an agent to process, not just human-legible. That’s what Headlight is built around.

It should open source soon. The team is already using it internally to track every token flowing through Headroom.

Conclusion

The three-stage pipeline holds together as an architecture. CacheAligner handles provider-level prefix caching mechanics that are easy to accidentally break. ContentRouter routes by content type rather than applying one strategy to everything. CCR handles the lossy-compression problem without requiring the application to change how it works.

The numbers in real sessions: 20–30% typical savings, 83–95% on high-redundancy data, same answers. Compounding with provider KV caching can reach 96% effective cost reduction with Anthropic at scale. Voice agent and industrial video use cases show the compressors work outside coding workflows.

The honest caveats: CompressBase is a first-generation model with room to grow. Enterprise proxy integration is unfinished. CCR’s 5-minute default TTL will catch you in long sessions. The accuracy evals are ongoing, not complete.

Test on your actual tool outputs before committing. Run the compression, check result.tokensAfter and result.transformsApplied. Lean outputs will see less benefit. 500-row JSON arrays or 1,000-line build logs are where it earns the integration cost.

Start with proxy mode: headroom wrap claude, check localhost:8787, and let the numbers decide.