About Virtual Context

Virtual Context is a context management system for LLMs built for long-running, tool-heavy conversations. It helps models keep the important parts of prior work available without forcing every request to carry the full raw history. The system segments conversation state by topic, compresses old material into recoverable memory, and retrieves the most relevant context when the model needs it.

The project is open source under the AGPL-3.0 license and is also available as a managed cloud product for teams that want hosted infrastructure, tenant management, billing, and dashboard tooling around the same core engine.

The problem Virtual Context was built to solve

LLM applications break down when conversations get long, when tools generate large volumes of output, or when the model needs to recall earlier decisions accurately. The standard approach of sending the full transcript on every turn becomes expensive, noisy, and eventually exceeds the context window entirely. Existing workarounds either throw away old context (sliding windows), retrieve fragments without preserving conversational flow (RAG), or simply hope that a larger context window will paper over the problem.

Virtual Context takes a different approach, inspired by how operating systems manage memory. Instead of choosing between keeping everything in RAM or losing it, an OS pages data in and out based on what the running process actually needs. Virtual Context applies the same principle to LLM context: it compacts older material by topic, maintains structured facts and summaries, and pages the most relevant memory back into the active prompt on demand. The result is a system where a model can recall decisions made thousands of turns ago while keeping the active context curated and cost-effective.

This makes the system a good fit for coding agents, research workflows, support and analyst tooling, and any application where context quality matters more than raw transcript size. The goal is not just a bigger prompt. The goal is better memory management for LLM behavior over time.

How it works

Virtual Context sits in front of any OpenAI-compatible API as a transparent proxy. Integration requires changing one base URL. No SDK rewrite, no new message format. It works with Anthropic, OpenAI, Gemini, Groq, Mistral, Together, and any other provider that exposes a chat completions endpoint.

Under the hood, the system runs a multi-stage pipeline on every conversation turn. The segmenter breaks conversation history into topic-coherent chunks. The compactor applies hierarchical compression, producing summaries and extracting structured facts (entities, preferences, decisions, relationships) that survive across compression tiers. The retriever uses a three-signal reciprocal rank fusion (recency, similarity, and keyword match) to select the most relevant memory for the current request. The assembler reconstructs the prompt, weaving together recent turns, retrieved summaries, structured facts, and system context within the model’s token budget.

Storage backends include SQLite for single-instance deployments and PostgreSQL for production multi-tenant setups. The proxy supports Redis session caching, streaming passthrough, multi-instance routing, and a live dashboard for monitoring conversations, segments, facts, and retrieval quality in real time.

Benchmark results

Virtual Context was evaluated against established long-context benchmarks using ICLR 2025 datasets. The system achieves 95% overall accuracy on LongMemEval, including 100% on knowledge-update questions that require tracking changing facts across long conversations. It delivers 2.2x fewer tokens per request compared to full-context baselines using Claude Sonnet, while maintaining or exceeding recall quality. Full methodology and results are published in the research paper.

Referenced by

Virtual Context has been referenced in independent analyses of LLM memory management and long-context benchmarking.

Your Agent Harness Owns Context, Not Truth by Mark Hendrickson (April 2026) examines how agent harnesses manage context windows and discusses Virtual Context as an external layer that compresses, indexes, and pages between memory tiers without controlling harness decisions. The article cites benchmark results showing 95% accuracy on LongMemEval compared to 33% for raw full-context baselines.

Proposal: A Real Benchmark for Long-Term AI Memory Systems by Penfield Labs (April 2026) presents a benchmark design for evaluating long-term memory in AI systems. The proposal references Virtual Context alongside other memory architectures when discussing how different systems approach context ingestion, and credits Ahmed Kidwai for architecture insights and corpus design feedback.

Project history

Virtual Context began as an investigation into why LLM agents degrade on long tasks. The core hypothesis was that the problem is not context window size but context management: models need curated, relevant memory, not raw transcripts. The first prototype implemented topic-aware segmentation and two-tier compaction on a single SQLite database, demonstrating that structured memory management could preserve recall quality while reducing token usage by more than half.

The project was released as an open source Python package on PyPI and a GitHub repository under the AGPL-3.0 license. The accompanying research paper, Virtual Context: Unbounded Context for LLM Agents via OS-Style Memory Management, was published on March 21, 2026, documenting the architecture, benchmark results, and design rationale.

The managed cloud product launched alongside the open source release, providing hosted infrastructure with tenant provisioning, API key management, billing, and a dashboard for teams that want the system without operating it themselves. Current pricing starts with a free tier supporting a 2M token window, a Pro plan at $19/month for a 10M window, and a Team plan at $99/month for 100M tokens with priority support.

Built by the project author

Virtual Context was created by Y. Ahmed Kidwai, who designed the architecture, wrote the engine, built the cloud platform, and authored the research paper. The work draws on practical experience with LLM systems in production, where the failure modes of naive context handling (lost recall, ballooning costs, degraded accuracy on long sessions) motivated a systems-level solution rather than a prompt-engineering workaround.

Ahmed maintains the open source project, operates the managed cloud product, and continues active development on the engine, proxy, and retrieval pipeline. He also writes about context management and LLM memory on the Virtual Context blog. Contributions, bug reports, and feature requests are welcome on GitHub.

To learn more, read the research paper, explore the documentation, review the source code on GitHub, or get in touch about enterprise deployment.