Proxy Deep Dive

Quick Start

The proxy sits between any LLM client and an upstream provider, enriching each request with retrieved context and capturing the response to build long-term memory. The proxy server ships in the base install.

pip install virtual-context

ANTHROPIC_API_KEY=sk-... virtual-context proxy \
  --upstream https://api.anthropic.com \
  --port 5757

# Point your client at http://127.0.0.1:5757 instead of the real API
# Dashboard: http://127.0.0.1:5757/dashboard

CLI Options

Passthrough vs Active Routing

Not every request takes the full pipeline. The proxy routes each request on whether the conversation has retrievable content: any compacted segments or indexed turns. A brand-new conversation has neither, so its requests pass through with minimal modification and near-zero added latency. Media compression still applies.

Once a conversation has retrievable content, requests take the active path. The gate is per-conversation and content-based, so cold starts stay cheap and enrichment begins exactly when there is something to enrich. Operators can also force a conversation into passthrough from the dashboard.

Request Lifecycle (active path)

1. Parse the POST body, detect the API format, extract the last user message.
2. Extract envelope: parse client channel metadata (sender, channel, reply target) and keep it as provenance, removing the wrappers from model-visible text.
3. Wait for the previous turn’s on_turn_complete to finish.
4. Reconcile the request history against stored canonical turns, appending anything new. On the first request this bootstraps the TurnTagIndex from the client’s existing history.
5. Enrich: on_message_inbound() tags the query, retrieves matching stored summaries, and assembles a context block.
6. Inject the <virtual-context> block into the system prompt or system message, on a deep copy of the body.
7. Forward upstream. Streaming uses raw byte forwarding to preserve exact SSE framing.
8. Capture the assistant text from streaming deltas or the response body.
9. Complete: fire on_turn_complete() on a background thread, which persists and tags the turn, updates the TurnTagIndex, checks compaction thresholds, and builds tag summaries.

Format Support

Four request formats are auto-detected per request, each with its own context injection point. Every pipeline stage is format-aware, one port handles all of them, and no configuration is needed.

Streaming

Raw SSE bytes are forwarded from the upstream to preserve exact framing, which some client SDKs depend on. A side-channel parser accumulates text deltas for on_turn_complete. Non-2xx responses such as rate limits and overloads are returned as JSON errors rather than broken SSE streams.

Diagnostics. Every streamed response emits three operator log lines: STREAM_FIRST_BYTE for time to the first upstream byte, STREAM_STALL for a mid-stream gap with the gap length and counters at that point, and STREAM_END with total elapsed time, chunk and byte counts and the largest gap observed. When a stream feels slow, these localize whether the delay was before the first byte, inside the stream, or downstream of the proxy.

Envelope Handling

Messages arriving from chat channels carry metadata that would pollute tagging if left in the model-visible text. The envelope parser first claims the useful parts, sender identity, channel and reply target, as provenance stored on the canonical turn, then strips the wrappers. Handled patterns include prompt markers, backward-compatible user wrappers, system event lines, channel headers and message-id footers.

Because sender identity is preserved rather than discarded, group participants appear as real names and timestamps give segments accurate chronological ordering.

Thread Safety

ProxyState holds a single-worker thread pool for background on_turn_complete work. wait_for_complete() blocks until the pending future resolves and is called at the start of each new request. History ingestion uses double-checked locking so the bootstrap runs exactly once.

Conversation Continuity

Conversation identity is derived from the system prompt and early messages, so the same client session routes to the same conversation across restarts. Sessions resume via durable storage; no client-side state is required.

Redis Session Cache

An optional write-through Redis cache persists conversation history and engine state across restarts, removing cold-start re-ingestion. It falls back to store-only operation when Redis is unavailable. Install with the redis extra.

Multi-Instance

One process can serve several providers on separate ports, each with its own engine and storage when given its own config file.

proxy:
  instances:
    - port: 5757
      upstream: https://api.anthropic.com
      label: anthropic
      config: ./vc-anthropic.yaml    # isolated engine + storage
    - port: 5758
      upstream: https://api.openai.com
      label: openai                   # shares the master engine

When proxy.instances is set, the --upstream flag is ignored.

Live Dashboard

A self-contained single-page application at /dashboard, with all CSS and JavaScript inlined and no external dependencies. It connects over Server-Sent Events for real-time updates.

Panels cover uptime, request, turn and compaction counters; a compaction memory bar with a manual compact control; average pipeline latency split into wait, inbound and injected context; cost savings against a simulated naive baseline; per-session segment and tag statistics; the active tag working set; a chronological request log of the last 200 requests with tags, tokens and latency; and a compaction event history.

A request inspector opens the original pre-enrichment payload for any logged request from a ring buffer of the last 50, colour-coded by role and downloadable as JSON. A settings modal exposes runtime-adjustable compaction, tagging, retrieval, assembly and summarization values with cross-field validation; changes apply to the running session and are not written back to the config file. Export downloads a full session snapshot for offline analysis.

Dashboard Auth

Dashboard endpoints, including the mutating ones, are unauthenticated by default. Set the VC_DASHBOARD_TOKEN environment variable to require a token; the server logs a warning at startup when it is unset.

The default bind address is 127.0.0.1, so the dashboard is not reachable from other hosts unless you change --host. If you bind a non-loopback address, set the token.

SSE Event Stream

The dashboard subscribes to GET /dashboard/events. On connection it receives a full snapshot of aggregate stats and recent events, then incremental events via a cursor-based sequence number.

Request Captures and Shared State

The request inspector is backed by per-turn request captures persisted to the store, keyed by conversation and turn and restored on startup, so captured payloads survive proxy restarts rather than living only in memory.

Conversation-scoped dashboard statistics are likewise saved as shared snapshots through the session-state provider. In multi-worker deployments every worker therefore serves the same dashboard state, regardless of which one handled the traffic.

Error Resilience

If the engine fails, the request is forwarded upstream unmodified. If an enriched payload comes out larger than the original, a bloat fallback reverts to the unmodified payload for that request. The proxy never blocks your LLM calls.

OpenClaw Plugin

The plugin uses lifecycle hooks rather than a bridge server: synchronous retrieval on the inbound hook and fire-and-forget compaction after the agent responds.