Observability

Telemetry is gated on RA_MCP_OTEL_ENABLED=true. When enabled, traces, metrics, and logs are exported via OTLP.

Environment Variables

Variable	Default	Description
`RA_MCP_OTEL_ENABLED`	`false`	Master switch
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`	Collector endpoint
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	`grpc` or `http/protobuf`
`OTEL_SERVICE_NAME`	`ra-mcp`	Service name (`service.name`)
`ENVIRONMENT`	(unset)	`deployment.environment.name` resource attribute (falls back to `DEPLOYMENT_ENVIRONMENT`)
`RA_MCP_OTEL_LOG_BRIDGE`	`true`	Bridge Python logging to OTel

The SDK sets service.name, service.version, a per-process service.instance.id, and — when ENVIRONMENT is set — deployment.environment.name as resource attributes; OTEL_RESOURCE_ATTRIBUTES is merged on top.

service.instance.id defaults to a fresh UUID per process so replicas are distinguishable (the chart autoscales to several pods). To pin a stable per-pod id, set it from the Kubernetes downward API:

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.instance.id=$(POD_NAME)"

Trace Tree

Every MCP tool call produces a trace spanning from the protocol layer down to HTTP:

graph TD
  A["tools/call search_transcribed\n<i>FastMCP — automatic</i>"]
  B["delegate search_transcribed\n<i>FastMCP — composed server</i>"]
  C["tools/call search_transcribed\n<i>FastMCP — provider</i>"]
  D["SearchOperations.search\n<i>domain layer</i>"]
  E["SearchClient.search\n<i>API client</i>"]
  F["HTTP GET\n<i>HTTP client</i>"]

  A --> B --> C --> D --> E --> F

Instrumented Components

Component	Tracer name	Spans	Metrics
HTTP client	`ra_mcp.http_client`	`HTTP GET`	request count, error count, retry count, duration, response size
Search client	`ra_mcp.search.client`	`SearchClient.search`	—
Search ops	`ra_mcp.search_operations`	`SearchOperations.search`	`ra_mcp.search.requests`, `ra_mcp.search.results`
Browse ops	`ra_mcp.browse_operations`	`BrowseOperations.browse_document`, `BrowseOperations._fetch_page_contexts`	`ra_mcp.browse.requests`, `ra_mcp.browse.pages`, `ra_mcp.browse.empty_pages`
ALTO client	`ra_mcp.alto_client`	`ALTOClient.fetch_content`	`ra_mcp.alto.fetches`
IIIF client	`ra_mcp.iiif_client`	`IIIFClient.get_collection`, `IIIFClient.fetch_manifest`	—
OAI-PMH client	`ra_mcp.oai_pmh_client`	`OAIPMHClient.get_metadata`, `OAIPMHClient.extract_manifest_id`	`ra_mcp.oai_pmh.fetches`
Viewer fetchers	fastmcp tracer (`fetchers.py`)	`fetch_text_layer`	—
LanceDB spine	`ra_mcp.lancedb`	`search <table>` (CLIENT) — all 13 datasets + PDF-guide search	`ra_mcp.lancedb.queries`, `ra_mcp.lancedb.errors`, `ra_mcp.lancedb.query.duration`, `ra_mcp.lancedb.results`
TORA geocoding	`ra_mcp.tora.client`	`tora sparql` (CLIENT)	—
PDF cache	`ra_mcp.pdf.cache`	`fetch pdf range` (CLIENT), `prefetch pdf` (INTERNAL)	—
CLI commands	`ra_mcp.cli.*`	`cli.search`, `cli.browse`	—

FastMCP adds automatic spans for all tools/call and resources/read operations. Handlers enrich the current tool-call span with behavioural intent (search terms via db.query.text, view.reference_code, pdf.url, geocode.place) and mark it ERROR via mark_span_error when they return an error string instead of raising — see Behavioural analytics for the product-question map.

Custom metrics all declare a UCUM unit (s, By, or an annotation such as {request} / {fetch} / {page}); metric dimensions are kept low-cardinality (method, status, result enums), while per-request identifiers (URLs, PIDs, reference codes, keywords, session ids) live on spans only.

Error recording

Failures set span.set_status(ERROR, "{ErrorClass}: …") and record error.type + a structured exception.* log record via record_span_exception() — the OTel Span Event API (record_exception) is deprecated, and the log record carries the active trace_id/span_id through the log bridge. The stacktrace is logged once per exception even as it unwinds through several layers (the counter isn't inflated). Retried-then-succeeded requests stay UNSET.

A tool handler that catches an exception (or fails input validation) and returns an error string instead of raising calls mark_span_error(message, error_type=…) first, so the FastMCP tools/call span goes ERROR — otherwise a normal return leaves it green and the tool failure rate reads zero.

Verify Telemetry

# Start Jaeger, exercise tools, check trace tree
dagger call verify-telemetry --source=.