You cannot debug what you do not log
LLM calls are non-deterministic, slow, and metered. When something goes wrong — a bad answer, a cost spike, a latency spike — you need a record of exactly what was sent and received. Structured JSON logs turn every LLM call into a queryable event, the same way structured logging transformed backend observability.
What to capture per call
{
"ts": "2026-06-17T10:00:00Z",
"trace_id": "req_8821",
"model": "claude-opus-4-8",
"operation": "classify_ticket",
"input_tokens": 412,
"output_tokens": 38,
"latency_ms": 740,
"cost_usd": 0.0091,
"temperature": 0,
"status": "ok",
"cached": false
}Each field answers a question you will eventually ask: which model, how slow, how expensive, did it succeed, was it cached, which request did it belong to.
Link calls to requests with a trace id
A single user action might trigger several LLM calls (retrieve, plan, answer). Tag them all with the same trace_id so you can reconstruct the full chain. This is the LLM equivalent of distributed tracing — it lets you see that a slow response was caused by the third of four model calls.
Track cost as a first-class metric
Token usage is money. Compute and log cost_usd per call from the provider's per-token pricing, then aggregate:
- By operation — which feature costs the most?
- By model — is the expensive model worth it over the cheap one?
- By tenant/user — who is driving spend (and should you rate-limit them)?
A dashboard built on these logs catches a runaway cost the day it starts, not at the end of the month.
What NOT to log
- Full prompt content with PII or secrets. Log a hash or a truncated preview, or redact sensitive fields — especially if your logs leave your infrastructure.
- Entire huge contexts. Log token counts and an id, not the whole retrieved document set.
- API keys. Never.
Balance debuggability against privacy: capture enough to diagnose issues without turning your log store into a liability.
Sampling and full capture
For high-volume systems, log metadata (tokens, latency, cost, status) for every call, but capture the full prompt and response only for a sample or for failures. That keeps storage manageable while preserving the detail you need when something breaks.