0 posts in this series
The operator's agent runtime runs on the workstation and owns identity and marker recovery. Everything past the LiteLLM socket is middleware that never knows who the operator is. The harness boundary is maintained by design: fail-open and CCR retrieval decoupling depend on it. CCR is lossy in transmission and lossless in recovery.
The operator’s agent runtime runs on the workstation and owns identity and marker recovery. Everything past the LiteLLM socket is middleware that never knows who the operator is. The harness boundary is maintained by design: fail-open and CCR retrieval decoupling depend on it. CCR is lossy in transmission and lossless in recovery.
CCR is a retrieval contract, not a cache. Headroom stores original content by hash and replaces it in transit with a marker; the original remains recoverable through headroom-mcp for 24 hours. Retrieval is split out of LiteLLM on purpose: only the harness owns the retrieve tool, so marker recovery cannot leak through the gateway.
LiteLLM is not a passive forwarder. Four custom plugin packages load from /config at import time; a pre-call guardrail compresses eligible requests; and post-call hooks reconcile savings against the route that served them. The shared litellm_call_id joins the request, compression outcome, and spend record.
The harness maintains its half of the middleware contract through extensions: session correlation stamps session_id and metadata.headroom_ccr; the CCR nudge teaches the model to retrieve markers; and CCR restore replaces OMP’s elided tool previews before the gateway compresses them. Shared libraries and the above-editor board make the contract observable and testable.
The inference stack is not all Docker. LiteLLM, Headroom, and GPU Kompress have separate image bases; HAProxy is configuration, not a container; and oMLX plus Mac Kompress run natively under launchd. The Kompress pool is mixed: GX10 GPU Docker workers and native Mac MPS workers sit behind Dell HAProxy, while the Dell Headroom container remains the proxy, not a worker.
The operator console is a read-only surface over the inference gateway. Two Postgres databases, one Valkey instance, and one LiteLLM HTTP API feed a single Next.js standalone app. The invariant: every page reads from data the middleware already wrote; the single write path is operator-initiated and goes to Valkey, not to the gateway or model.
OMP skills are not hand-written in a single tree. They are assembled from upstream repositories by a sync script that sparse-checks out subpaths, overlays local private patches, and validates frontmatter before the skill becomes visible to the agent. This post derives the three-stage pipeline and the committed-vs-generated boundary that makes it work.
The pytest test suite under infra/home-infrastructure/groups/inference/litellm/tests/ is the executable specification of the middleware’s contracts. 14 primary test files across 6,500+ lines exercise every admission gate, breaker state, CCR kill-switch, credit-failover path, and savings-reconciliation invariant. If a middleware behaviour is not enumerated as a test case, it is not part of the contract.