Evidence retrieval (CCR)

Evidence retrieval — Observability Unified's compressed context retrieval (CCR) layer — turns raw observability data into compact, agent-sized evidence. Instead of an agent expanding a whole trace and every correlated log into its context window, the collector returns a small evidence bundle: the critical-path summary, the failed-span anchor, a few log exemplars, and retrieval refs the agent can expand on demand.

The goal is the same symptom → evidence → root cause loop, but within a token budget: keep the debugging signal, drop the redundancy.

Why it exists

A single failing checkout trace can carry hundreds of correlated, near-identical logs. Pasting all of that into an agent is expensive and drowns the signal. CCR compacts repeated records into exemplars, preserves the findings that matter (e.g. the failed payment.authorize span), and hands back references the agent can expand only if it needs the raw rows.

MCP tools

The MCP server exposes four read-only tools for this layer:

• get_evidence_bundle — compact evidence for a trace, action, agent_run, tool_call, change, finding, or incident anchor, scoped to an investigation intent (debug_failure, triage_incident, explain_latency, explain_cost, inspect_agent_run, inspect_tool_call, find_instrumentation_gap, or general) and a token budget.
• retrieve_evidence_ref — expand a bundle retrieval ref into raw, less-compacted, or redacted records. chunkOffset paginates replay event windows.
• search_evidence_ref — search within a log retrieval ref without expanding the full evidence slice.
• get_evidence_stats — issued/expanded evidence-ref telemetry for the active project.

Bundles and refs use the EvidenceReference contract (@obsunified/types), so confidence, relationshipCertainty, causalProofTier, citations, and suggestedNextPivots carry through. Retrieval is project-scoped: replay event windows and profile frame summaries require explicit expansion, AI raw request/response data stays redacted, and tool expansion returns hashes plus redacted args/results.

What a bundle preserves

• The critical-path summary for the anchor.
• Failed-span evidence (the debugging anchor is never compacted away).
• Log exemplars with compaction provenance (how many records collapsed into each).
• Causal context for action/agent-run/tool-call anchors: side-effect, approval, eval, trace, and log links.
• For triage_incident and AutoSRE anchors (change, finding, incident), bundle.autosre separates:
  • faultLocus ({ nodeId, spanId, relationshipCertainty, isBlackoutSuspected } — the deepest error span; relationshipCertainty is link provenance, and locating a span carries no causal proof on its own)
  • candidateChanges (corroborated_candidate / cohort_rate_isolated / counterfactual_verified, with change blast-radius metrics)
  • contentionCandidates (noisy-neighbor traces hitting the same downstream service within ±30s)
  • priorFindings (historical InvestigationFindings matching the fault locus entity id or the same recurrence fingerprint — service, route, and error signature — in the same project)

Reduction, measured

CCR keeps its product claim executable. The repo ships a deterministic benchmark (pnpm benchmark:ccr) that runs the bundle path against a checkout failure with 500 correlated 404 logs and asserts the bundle still compacts and still keeps the failed-payment anchor. On the reference scenario it reduces a ~50k-token raw context to ~1.3k tokens (~97% fewer tokens) while preserving the failed-span finding. See docs/benchmarks/evidence-retrieval-ccr.md for the current numbers and methodology — re-run the benchmark before changing any CCR reduction claim.

Instrumentation gaps

Every trace read also reports where instrumentation is probably missing. The collector computes these gaps at read time from the trace's spans: a span is flagged when it runs longer than 100 ms, more than 70% of its duration is its own time rather than its children's, it has at most one child span, it is not an async parent, and it is not a client, producer, or consumer span (where waiting on another system is expected).

• The trace waterfall in the dashboard marks each flagged span with how many milliseconds are unaccounted for and suggests adding child spans or attaching a profile.
• The MCP get_trace tool and GET /internal/telemetry/traces/:traceId return the gaps as instrumentationGaps (each gap's span, uninstrumented time, and share of its parent), and GET /internal/telemetry/traces/:traceId/gaps returns them alone.
• Analyses that carry gaps turn them into evidence references with the uninstrumented_self_time_gap predicate, so agents can cite them.
• GET /internal/telemetry/instrumentation-gaps/calibration scans recent traces (72 hours by default) and reports how often gaps occur, so you can check the thresholds against your own traffic.

The thresholds are heuristics. A flagged span can be legitimate CPU work, so treat a gap as a place to look, not as proof of missing spans.

Related

• Evidence reference — the EvidenceReference contract bundles and refs are built on.
• MCP server — the read boundary that exposes these tools.
• RFC 0011 — Evidence retrieval layer — the design.