AI Observability: What to Log, What to Ignore
A working schema for logging LLM traces, tool calls, cost and quality — plus what most teams get wrong.
The trace is the primary unit
For LLM systems, the log line is the trace — the full sequence of prompts, tool calls, model versions, tokens, latencies and outputs for one user request. Everything else derives from it. Design your logging around traces, not lines.
Log these fields always
Request ID, user ID (hashed), tenant, model name + version, prompt template ID, retrieved doc IDs, tool call names + args, token counts (in/out), cost, latency per stage, and final output. On failures, add the exception and any tool-response payloads. This gives you replay, debugging and cost attribution for free.
What to sample vs log fully
Log every trace's metadata (fields above). Sample 1–10% of full prompt/output payloads for eval and debugging; log 100% on flagged errors or user thumbs-down. Full logging on 100% burns storage without proportional insight.
Dashboards that pay for themselves
Cost per user per day. Latency p50/p95/p99 per pipeline stage. Retrieval hit rate on user thumbs-down. Model version regression tracking. Tool error rates per tool. These five dashboards catch most production LLM issues before customers report them.
Privacy and retention
PII redaction on ingest for logs. Short retention (30–90 days) for full payloads; longer for aggregated metrics. Never log secrets, auth tokens or raw customer content beyond retention windows. Legal/compliance sign-off before enabling full-payload logging in regulated sectors.
Key takeaways
- The trace is the primary unit; design logging around it
- Always log metadata; sample 1–10% of full prompt/output payloads
- Five dashboards catch most issues: cost/user, latency, retrieval hit rate, model regression, tool errors
- PII redact on ingest; short retention on full payloads
- Log 100% on flagged errors and user negative feedback
T7 Research
Enterprise AI Research Group
T7 Research is the research arm of T7 Solution, focused on benchmarking LLMs, evaluating RAG patterns, and compiling implementation playbooks for enterprise technology leaders.