Monitor AI Call Latency and Error Rates

Define the AI call you want to measure

1. Start with one application workflow, such as generating a support reply. Define a logical call as the complete operation your application requests, including any retries. Track individual network attempts separately so your dashboard can show both the user experience and the underlying reliability.

Write down what successful completion means. For a structured response, require both a completed request and a valid result. Record schema validation failures separately from transport failures, and keep answer-quality evaluation outside your basic availability metric.

Capture timing around the complete response

2. Place a timer immediately before the operation begins and stop it when processing finishes or fails. Use a monotonic clock, which is intended for measuring elapsed time. For streaming calls, record time to first chunk separately from total duration; OpenTelemetry defines distinct measurements for these two concepts. ([github.com](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/metrics.yaml?utm_source=openai))

Decide whether your first-chunk measurement means any received chunk or the first visible content. Document that choice. Keep timing until the stream ends, rather than stopping when response headers arrive, and record an interrupted stream as an incomplete operation.

Record useful metadata without capturing sensitive content

3. Emit one structured event per attempt and one summary event per logical call. Include a timestamp, trace identifier, requested model, streaming setting, elapsed time, outcome, and attempt number. Add the returned model identifier and token usage when available.

Keep prompts, completions, credentials, and personal information out of routine monitoring events. Put request identifiers in logs or traces rather than using them as metric labels. Start with a small, predictable label set such as application, environment, model, and outcome.

Separate API errors from unsuccessful user operations

4. Build explicit categories for authentication problems, invalid requests, throttling, server errors, connection failures, timeouts, cancellations, and incomplete streams. Preserve the provider's error code when available. OpenAI's error documentation distinguishes these conditions and gives different recovery guidance for them. ([developers.openai.com](https://developers.openai.com/api/docs/guides/error-codes?utm_source=openai))

Calculate attempt error rate as failed attempts divided by all attempts in the same window. Calculate logical-call failure rate separately, using operations that ultimately failed divided by all logical calls. Show cancellations separately and state whether your headline failure rate includes them.

Build a dashboard that exposes slow requests

5. Add panels for call volume, logical-call failures, attempt errors, retries, and latency. Use a duration histogram to calculate p50, p95, and p99 rather than relying only on an average. OpenTelemetry's generative AI conventions define client operation duration as a histogram measurement. ([github.com](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md?utm_source=openai))

Show streaming first-chunk timing alongside completion timing. Compare like-for-like workloads: keep streaming settings, output limits, and prompt sizes reasonably consistent. For example, give Gemini 2.5 Flash its own model filter rather than mixing every model into one latency chart. Display sample counts beside percentiles.

Test retries and alerts before relying on them

6. Exercise the monitoring path with controlled failures, including a timeout and an interrupted stream. Inspect your SDK's retry settings before adding another retry layer. OpenAI's JavaScript SDK automatically retries certain connection failures and HTTP errors, so an application wrapper must account for that behavior. ([developers.openai.com](https://developers.openai.com/api/reference/typescript?utm_source=openai))

Use bounded retries, an overall deadline, and backoff with jitter where appropriate. Honor Retry-After when supported and applicable; OpenAI recommends this approach for rate-limit recovery. ([help.openai.com](https://help.openai.com/en/articles/5955604-troubleshooting-api-rate-limits-and-429-errors?utm_source=openai)) Set initial alerts from your application's response-time budget, then tune them against observed traffic. Require sufficient volume and a sustained breach before paging someone.

Evaluate a relay with the same measurements

7. If you want pay-as-you-go access across multiple model families, Relay365 is worth evaluating as a relay desk. Instrument your application around the complete relay request, and compare equivalent workloads before changing routing. Treat your measurements—not the presence of a relay—as evidence of performance.

Create a Relay365 account when you are ready to run that comparison. Begin with a small workload, inspect latency and failure patterns, and expand only after your dashboard explains both successful calls and failures.