An agent’s final answer is often the least useful part of its production trace. Operators need to reconstruct which model was invoked, what it was asked to do, which tools ran, whether a reviewer approved an action and how long each step took. OpenTelemetry’s generative-AI conventions provide vocabulary for those events, but instrumentation must protect secrets and respect evolving semantic versions.
Trace a complete request, not just tokens
Create an end-to-end trace ID at the authenticated request edge. Child spans should cover retrieval, model invocation, tool execution, approval waiting, queue processing and final outcome reconciliation. Link resumed jobs to the initiating request even when work outlives the initial HTTP call. Record timestamps, selected model or tool version, number of attempts, error class and sanitized destination identifier. Mark the difference between a tool proposal and an externally observed side effect.
Conventions and version drift
OpenTelemetry documentation defines GenAI-related attributes and operations including tool invocation. The semantic conventions evolve and some fields have development status, so pin the specification revision used in dashboards and collectors. Test instrumentation changes before deploying a new model SDK; a span name can change without the underlying workflow changing. Normalize only what is stable and preserve provider-specific metadata in separate namespaced fields.
Protect content inside traces
Prompts, retrieved paragraphs, authentication headers and tool results can contain credentials, medical details or personal identifiers. Default to structured metadata rather than raw payload capture; use allowlists and targeted redaction before export. Avoid logging vector-search excerpts or whole browser screenshots without an explicit need and retention rule. Make debug capture short-lived, access-controlled and disabled by default in high-risk workflows. Treat your telemetry backend as another sensitive data processor.
A practical operator dashboard
For an agent that drafts customer-support responses, show completion count, external sends actually confirmed, tool failure rates, first response latency, end-to-end wall time, cost allocation by tenant and approval queue depth. A time series of model tokens alone misses slow browser loads or human delays. Preserve correlation from the draft to the final sent message ID and policy evidence. Alert on missing expected reconciliation spans as well as overt exceptions.
Validate the instrumentation
Inject a controlled tool failure, an auth denial, a duplicate callback and an expired approval. Confirm that the trace identifies the failing stage without exposing the payload. Compare span counts to the execution ledger; sampling should not erase critical audit events. Choose privacy and operational correctness before completeness of textual traces. No source here represents a vendor-independent benchmark of a specific observability platform.
Choose diagnostic granularity deliberately
A production team debugging an agent needs to know whether latency came from retrieval, model inference, rate limiting, browser navigation, approval waiting or the target API. Tool-level spans and a durable transaction ledger answer different questions; do not substitute a sampled trace for a legally significant audit entry. Keep service-level objectives tied to real user outcome and failure severity, not only response-token throughput.
Production acceptance and review test
Compare sanitized trace IDs with server-side tool audit IDs under a deliberately lost response, duplicate callback and delayed human approval. Verify that investigators can reconstruct the sequence without access to plaintext credentials or document bodies. Check who can query the trace backend and the backup retention policy. Any instrumentation upgrade should be tested for new field names and accidental raw prompt capture.
Scope, version and availability
Features documented by a current vendor page are not assumed to have existed in earlier releases. Where a historical launch date is provided, it is supported by a dated first-party publication. Exact subscription tier, market availability, quotas, API billing, licensing permissions and deployment regions are not asserted without a directly applicable record. This is researched editorial guidance rather than a claim of independent hands-on testing, deployed operation or statutory certification.
Sources and Further Reading
- OpenTelemetry — GenAI semantic conventions for agent spans (source reviewed 2026-10-25)
- OpenTelemetry — GenAI tool and model spans (source reviewed 2026-10-25)