Agent Observability: Traces, Audit Logs and What They Cannot Prove

Evidence check: 25 October 2026. Product details reflect the referenced documentation observed at this cutoff; exact plans, prices, regional eligibility and unverified release dates are not claimed. This is an editorial guide, not a hands-on test.

Three layers of operational evidence

A trace records a workflow or request and its spans: model calls, tool calls, handoffs and timing. An audit log records security-relevant events such as credential use, authorization, approvals and writes. A verification record captures the external result of a task, such as a transaction reference or a retrieved record version. These serve different purposes. A trace with a successful tool span is not automatically proof that a business transaction committed.

What to capture

Include a stable run ID, parent workflow ID, actor, tool name, sanitized argument hash, response status, duration, token or usage counters, model or workflow version and checkpoint state. Correlate retries with the original operation. Use separate tags for success, failure, cancellation and indeterminate outcomes. Keep personally identifying source content out of analytics unless there is a justified retention policy.

Tracing products and limitations

OpenAI’s Agents SDK documents built-in tracing of tool calls, guardrails and handoffs; its documentation also notes constraints for Zero Data Retention organizations. Arize Phoenix describes traces, datasets and evaluations used to inspect failures and compare changes. Documentation does not establish that any specific application is configured correctly. An implementation must validate export destination, redaction, retention and operator access.

Troubleshooting a failed task

Start from the reported outcome and traverse backward: was a tool invoked, with which approved parameters, by which principal, and what response returned? Did the workflow retry? Did a human alter the payload? Was the failure model-level, retrieval-level, connector-level or policy-level? This method narrows an incident without exposing raw transcripts to every employee.

Metrics worth tracking

Track completion verified by external state, policy denials, intervention rate, retries per operation, time to resolution, token consumption and cost per completed task. A low refusal rate can signal insufficient enforcement rather than good performance. Compare on matched task cohorts, not aggregate popularity alone.

Privacy and security trade-offs

Traces themselves can contain sensitive prompts and retrieved documents. Minimize fields, redact before exporting, secure telemetry storage and set retention. Allow incident investigators to access detailed events without granting routine dashboard users unrestricted view of customer content.

Trace one concrete request

For each run, capture a correlation ID, authenticated role, selected workflow version, model version identifier when known, tool name, argument validation result, tool latency, retry count, approval ID and external operation ID. Separate model-generated rationale from verifiable actions. A reviewer should be able to reconstruct why a file was read or a draft was changed without storing an entire unredacted conversation in routine logs.

Design retention around sensitivity

Trace systems may ingest prompts, search results, attachments or personally identifiable information. Set retention and redaction rules before enabling automatic tracing in production. Restrict who can see logs, secure vendor export destinations and record whether a trace is a sampled diagnostic or complete audit event. An observed “tool succeeded” span is not equivalent to checking final external state; use a separate reconciliation record. Measure instrumentation overhead because tracing itself may affect latency and storage cost.