Voice AI Agent Observability: 5 Failures w/ OTel

Core

• Most Voice AI agents fail in production because teams have 0 observability across telephony, streaming audio, and LLM tool calls.
• This post shows how to implement voice AI agent observability with OpenTelemetry by separating infra failures from conversation failures.
• You’ll build a Docker Compose lab to reproduce failures, validate signals, and turn them into actionable alerts before the next 300K calls.

Introduction

Voice AI agents are a distributed system disguised as a single “bot.” A single call can traverse telephony ingress, a streaming audio pipeline (ASR), an LLM, tool/function calls, a TTS stream, and then back out over a real-time connection. When something goes wrong, the user only experiences “the agent is broken,” while your dashboards often show a green CPU graph.

The production failure pattern is consistent: teams scale to tens or hundreds of thousands of calls, then discover they can’t answer basic questions like “where did latency come from?”, “was it the model or the network?”, or “why did the agent say that?” That’s the 0 observability trap.

This post focuses on LLM observability for voice agents using OpenTelemetry as the instrumentation layer. You’ll learn five common failure modes, how to separate infrastructure failures from conversation failures, how to design a tracing/logging map that supports debugging, how to reproduce issues locally with Docker Compose, and what to alert on before your next 300K calls.

1) The “0 observability” trap: why Voice AI agent stacks fail in production

“0 observability” doesn’t mean you have no monitoring. It means you have monitoring that can’t explain user-visible failures. In voice, that gap is wider because the system is real-time and multi-modal (audio + text + tool calls).

In real-time voice systems, a 300–500ms regression is often the difference between “natural” and “unusable,” and without end-to-end traces you can’t attribute that regression to ASR, LLM, tools, or TTS.

Failure mode #1: Latency compounding across hops

Voice stacks often have “acceptable” latency per component, but the user experiences the sum: jitter buffers, ASR partials, LLM time-to-first-token, tool round trips, TTS synthesis, and network egress. Without traces, teams guess and “optimize” the wrong hop.

Failure mode #2: Streaming breaks that look like model failures

A dropped websocket, a stalled gRPC stream, or backpressure in an audio pipeline can manifest as “LLM stopped responding.” If you only track LLM error rate, you’ll blame the model while the real issue is transport or buffering.

Failure mode #3: Tool-call flakiness and hidden retries

Many agents call tools (CRM lookup, calendar, ticketing) mid-conversation. Silent retries, timeouts, and partial failures can cause long pauses or contradictory responses. If tool spans aren’t linked to the call session, you can’t prove causality.

Failure mode #4: Prompt/turn bugs that don’t show up in infra metrics

Bad turn segmentation, missing conversation state, or prompt regressions can cause hallucinations or policy violations while CPU/memory and HTTP 200s look fine. You need conversation-level signals, not just infrastructure signals.

Failure mode #5: Cost blowups and token leaks

Voice agents can accidentally send huge contexts every turn, double-call the LLM, or loop on tool calls. You’ll see a bill spike after the fact unless you track token usage, turns per call, and tool-call rates per session.

2) Separate infrastructure failures vs conversation failures (stop one-score dashboards)

A single “health score” dashboard is attractive but misleading. Voice AI failures split into two categories with different owners, mitigations, and alert thresholds:

• Infrastructure failures: transport, timeouts, saturation, dependency outages, stream stalls, queue backlogs.
• Conversation failures: wrong intent, bad tool choice, hallucination, policy breach, user frustration, excessive silence, repeated questions.

What to measure for infrastructure failures

• End-to-end call latency breakdown (p50/p95/p99) by hop: ASR, LLM, tool, TTS.
• Streaming health: reconnect rate, audio chunk backlog, dropped frames, time-to-first-audio.
• Dependency SLOs: tool API error rate, timeout rate, and tail latency.
• Resource saturation: CPU throttling, memory pressure, event-loop lag (if applicable), connection pool exhaustion.

What to measure for conversation failures

• Turns per call, average silence duration, interruptions/barge-ins, and “reprompt” counts.
• Tool-call correctness proxies: tool-call rate per intent, tool-call retries, tool-call success per tool.
• Safety/policy outcomes: blocked responses, escalation-to-human rate, and user hang-up rate after specific agent actions.

Make the split explicit in your UI

Instead of one score, create two top-level views:

• Call reliability (infra): “Can the system complete calls with acceptable latency?”
• Call quality (conversation): “Did the agent accomplish the task without degrading UX?”

3) Build a practical tracing + logging map for Voice AI agent debugging

The goal is simple: for any failed call, you should be able to open one trace and answer: what happened, where did time go, and what did the agent decide? This section describes a pragmatic map you can implement incrementally.

Trace model: one trace per call session

Create a trace per call with a stable identifier (e.g., call_id) and attach it everywhere. Within that trace, create spans for each hop:

• telephony.ingress (call start, connection established)
• asr.stream (audio in → partial/final transcripts)
• llm.turn (per user turn; include model name, tokens, latency)
• tool.call (per tool invocation; include tool name, status, retry count)
• tts.stream (text → audio out; time-to-first-audio)
• telephony.egress (audio delivered, call end)

Logging model: structured events with correlation IDs

Logs should be structured JSON and include trace_id, span_id, call_id, and turn_id. Don’t log raw audio. For text, log safely: redaction, hashing, or sampling depending on compliance requirements.

Metrics model: a small set of high-signal counters and histograms

Metrics are for alerting and trend analysis. Keep them small and stable:

• call_duration_seconds (histogram)
• llm_latency_seconds, tts_latency_seconds, tool_latency_seconds (histograms)
• tool_errors_total, call_failures_total (counters)
• turns_per_call, reprompts_total (histogram/counter)

OpenTelemetry collector: the hub for traces/logs/metrics

In practice, you want apps to emit OTLP to a local or production OpenTelemetry Collector, then route to your backend(s). Below is a working collector config that accepts OTLP over HTTP/gRPC and exports traces to Jaeger, metrics to Prometheus, and logs to a file (good enough for a lab).

Create otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:9464
  file:
    path: /var/log/otel/voice-agent.log

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [file]
  

The featured technology’s role: OpenTelemetry as the correlation layer

In a voice agent, the hardest part is not collecting some logs; it’s correlating events across components and time. OpenTelemetry provides the shared context propagation and the data model (spans/log records/metrics) so you can:

• Follow a single call_id across ASR → LLM → tools → TTS.
• Attribute latency to the correct hop and see tail latency.
• Join “conversation failures” (reprompts, hangups) to the infra events that preceded them (timeouts, retries).

4) Docker Compose lab: reproduce failures and validate monitoring signals

You don’t need a full telephony provider to validate your observability design. A local lab can simulate the same failure modes: timeouts, retries, and latency spikes. The key is that every simulated “call” emits traces/metrics/logs through OTLP so you can verify correlation in Jaeger and Prometheus.

Lab topology

• voice-agent: a small Python service that simulates call sessions and emits OTel telemetry.
• tool-service: a dependency that sometimes fails or slows down.
• otel-collector: receives OTLP and exports to Jaeger/Prometheus/file.
• jaeger: trace UI.
• prometheus: metrics scraping.

Docker Compose: run the whole stack locally

This Compose file wires the services together on a shared Docker network, exposes Jaeger and Prometheus, and mounts the collector config.

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.106.1
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro
      - otel-logs:/var/log/otel
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
      - "9464:9464"  # Prometheus exporter

  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "16686:16686"  # UI
      - "14250:14250"  # gRPC ingest

  prometheus:
    image: prom/prometheus:v2.53.1
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "9090:9090"

  tool-service:
    build:
      context: ./tool-service
    environment:
      - PORT=8081
    ports:
      - "8081:8081"

  voice-agent:
    build:
      context: ./voice-agent
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - TOOL_URL=http://tool-service:8081
      - SERVICE_NAME=voice-agent
    depends_on:
      - otel-collector
      - tool-service

volumes:
  otel-logs:
  

Prometheus scrape config

Prometheus scrapes the collector’s Prometheus exporter endpoint. Create prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: otel-collector
    static_configs:
      - targets: ["otel-collector:9464"]
  

Minimal Dockerfile for the simulated voice agent

This Dockerfile builds a small Python service that emits OTel spans and metrics over OTLP HTTP.

FROM python:3.12-slim

WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

RUN pip install --no-cache-dir \
    fastapi==0.115.0 \
    uvicorn[standard]==0.30.6 \
    requests==2.32.3 \
    opentelemetry-api==1.27.0 \
    opentelemetry-sdk==1.27.0 \
    opentelemetry-exporter-otlp==1.27.0 \
    opentelemetry-instrumentation-fastapi==0.48b0 \
    opentelemetry-instrumentation-requests==0.48b0

COPY app.py /app/app.py

EXPOSE 8080
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
  

Run the lab and generate failures

These commands build the images, start the stack, and generate a burst of simulated calls so you can inspect traces in Jaeger and metrics in Prometheus.

# Start the lab
docker compose up --build -d

# Tail collector logs (you should see OTLP traffic)
docker compose logs -f otel-collector

# Generate 50 simulated calls (each call emits spans + metrics)
for i in $(seq 1 50); do
  curl -s -X POST http://localhost:8080/simulate_call \
    -H 'content-type: application/json' \
    -d '{"turns": 6, "tool_failure_rate": 0.15, "tool_p95_ms": 900}' >/dev/null
done

# Open UIs:
# Jaeger:      http://localhost:16686
# Prometheus:  http://localhost:9090
  

What to look for in the traces

• One trace per call with spans for each turn and tool call.
• Tail latency: a small percentage of calls dominated by tool.call or llm.turn.
• Errors: tool timeouts should appear as span status errors and increment error counters.

5) Production runbook: what to alert on before the next 300K calls

Once you have correlation, the next step is operationalizing it. A production runbook should define: (1) what “bad” looks like, (2) how you detect it early, and (3) what you do in the first 10 minutes.

Alerting: infra (call reliability)

• End-to-end p95 latency above threshold for N minutes (and broken down by hop).
• Streaming stall rate: calls with no outbound audio for > X seconds.
• Dependency timeouts for tools (rate and p95/p99 latency).
• Call failure rate (dropped calls, session init failures, unexpected disconnects).

Alerting: conversation (call quality)

• Reprompt spike: “Sorry, I didn’t catch that” events per call rising.
• Turns-per-call spike: indicates loops or inability to complete tasks.
• Hangup-after-agent-response spike: proxy for frustration or policy issues.
• Tool-call anomaly: sudden increase in tool calls per turn or repeated tool retries.

First 10 minutes: a deterministic triage flow

1. Pick a failing call (recent, user-reported or sampled from error budget burn).
2. Open the trace and identify the dominant span by duration (ASR vs LLM vs tool vs TTS).
3. Check error tags on spans: timeouts, HTTP status, retry count.
4. Correlate to metrics: is this systemic (p95 shift) or isolated (single dependency)?
5. Mitigate: disable a flaky tool, reduce tool timeout, switch to a fallback response, or shed load.

Deploying the instrumented service to Kubernetes (image build/push/deploy)

If your voice agent runs in Kubernetes, treat observability as part of the release. Build and push the instrumented image, then deploy with OTLP env vars pointing to your collector.

# Build and push (example uses Docker Hub)
export IMAGE=docker.io/yourorg/voice-agent:0.1.0

docker build -t $IMAGE ./voice-agent

docker push $IMAGE

# Deploy (assumes you already run an OpenTelemetry Collector in-cluster)
kubectl -n voice apply -f k8s/voice-agent-deployment.yaml

# Verify rollout
kubectl -n voice rollout status deploy/voice-agent
  

Example Kubernetes deployment manifest (OTLP wired)

This manifest shows the minimum env vars to export OTLP telemetry to a collector service. Save as k8s/voice-agent-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: voice-agent
  namespace: voice
spec:
  replicas: 3
  selector:
    matchLabels:
      app: voice-agent
  template:
    metadata:
      labels:
        app: voice-agent
    spec:
      containers:
        - name: voice-agent
          image: docker.io/yourorg/voice-agent:0.1.0
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_SERVICE_NAME
              value: voice-agent
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector.otel.svc.cluster.local:4318
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: http/protobuf
            - name: OTEL_TRACES_EXPORTER
              value: otlp
            - name: OTEL_METRICS_EXPORTER
              value: otlp
            - name: OTEL_LOGS_EXPORTER
              value: otlp
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: voice-agent
  namespace: voice
spec:
  selector:
    app: voice-agent
  ports:
    - name: http
      port: 80
      targetPort: 8080
  

Turn your five failure modes into concrete alerts

Here’s how the earlier failure modes map into signals you can actually page on:

• Latency compounding → alert on end-to-end p95 plus per-hop p95 (to localize).
• Streaming breaks → alert on “no outbound audio > Xs” and reconnect spikes.
• Tool flakiness → alert on tool timeout rate and tool p99 latency per tool name.
• Prompt/turn bugs → alert on reprompt rate, turns-per-call, and hangup-after-response spikes.
• Cost blowups → alert on tokens-per-call (or proxy: LLM requests per call) and tool calls per turn.

Conclusion

After enough production volume, Voice AI agent failures stop being “bugs” and become “unknown unknowns” unless you can correlate a single call across ASR, LLM turns, tools, and TTS. The fix is not another one-score dashboard; it’s a split-brain approach: call reliability (infra) and call quality (conversation), both tied together by end-to-end traces.

If you implement voice AI agent observability with OpenTelemetry as shown here—one trace per call, structured logs with correlation IDs, and a small set of stable metrics—you’ll be able to reproduce failures locally, debug them deterministically, and alert on leading indicators before the next 300K calls. Next step: stand up the Compose lab, validate your hop-by-hop latency breakdown, then promote the same instrumentation to production with a collector-backed pipeline.