The problem: your dashboards are green, the answers are garbage
Your inference pipeline responds with 200 OK. Latency P99: 420ms. Costs within budget. But the customer reports the model is "hallucinating product names that don't exist." Traditional monitoring doesn't see it. ML observability is not APM.
In 2026, the gap between "the service responds" and "the response is useful" defines whether your ML product scales or turns into expensive technical debt.
Why traditional APM fails with ML
- Semantic failures, not crashes: An LLM can return valid, syntactically correct JSON that is factually wrong. There is no stack trace.
- No real determinism: Even with
temperature=0, floating-point variation and provider sampling produce up to 15% accuracy variance (Trautmann et al., 2024). - Multi-step causality: In 22-step agents, a failure in the final output often originates at step 3. Call-level tracing can't see it.
Rule of thumb: If you can't reconstruct the full session trace (retrieval → tool calls → reasoning → output) when a user reports an error, your observability is incomplete.
Three observability layers you need
1. Full session-level tracing
# What you should capture on every request:
- Exact prompt sent (system + few-shot + user + tool schemas)
- Exact completion returned
- Retrieval chunks with scores (RAG)
- Tool call sequence: args + responses
- Injected memory context
- Tokens in/out, model ID, computed USD cost
OpenTelemetry GenAI semantic conventions (2026, Development stability) now standardize span names: chat, embeddings, execute_tool. Instrument at your capture layer, transport via OTLP collector, visualize in a swappable backend. Don't couple your code to the vendor SDK on day one.
2. Anti-drift baselines: three dimensions
Drift arrives silently. API 200, latency OK, costs OK. But answers lose grounding, change tone, fabricate strategies. Three baselines, per FutureAGI 2026:
- Input-distribution drift: Cosine distance between your golden set centroid and a rolling 7-day live-traffic centroid. >2σ off the 30-day baseline = alert.
- Prompt-template drift: The silent killer. A dev changes the system message or few-shots without re-baselining. Hash the template + auto-diff in CI.
- Retrieval-corpus drift (RAG): BM25/embedding overlap against a dataset baseline. Silent re-chunking rotates top-k.
3. Inline evaluation (not nightly batch)
Average LLM latency: ~647ms. Eval overhead: +150ms (23%). LLM-as-judge GPT-4-class: +1000ms+ (doubles latency). The 2026 solution: purpose-built Small Language Models (SLMs) that run 10-20 metrics in parallel in <200ms and block/transform/route unsafe outputs before they ship.
# Recommended pipeline:
Request → LLM → SLM-eval (groundedness, faithfulness, policy)
→ if PASS: return to user
→ if FAIL: fallback / rewrite / escalate to human
RAG needs its own eval layer: Retrieval quality (Precision@k, Recall@k) and Generation quality (groundedness, faithfulness) are independent systems. Both run against a maintained golden query set. A routine index refresh can tank answer quality without you noticing.
Inference FinOps: the metric almost nobody watches
Traditional services bill by compute time. LLMs bill per token, and tokens are wildly uneven across users. A pathological 1% of requests can eat 50% of your budget without triggering a latency alert.
Five signals per call (OpenObserve 2026): input tokens, output tokens, total tokens, model ID, computed USD.
The most useful derived metric: P99/P50 cost ratio. If your P99 request costs >50x your median, something is very wrong. Usually it's unconstrained max_tokens on a high-traffic endpoint. Fix that and your cost curves flatten dramatically.
Semantic caching and model routing: cutting costs 30-70%
- Semantic caching: Eliminates 30-70% of redundant API calls by matching paraphrased queries, not just exact strings. Different from native prompt caching (Anthropic 90% input tokens, OpenAI automatic).
- Semantic / model routing: Classify simple requests → small/cheap model; complex → flagship model. Can cut inference costs in half (TechTarget 2026).
- Prompt compression + response caching: Gorilla Logic 2026: from $12M/mo to $450K/mo combining a 25x cheaper model + semantic caching + prompt compression.
Recommended 2026 stack (vendor-neutral)
| Layer | OSS/Self-host option | When to use it |
|---|---|---|
| Capture | OpenTelemetry + GenAI conventions | Always. Day one. |
| Transport | OTLP Collector | Always. Decouples the backend. |
| Inline eval | OpenLIT / Arize Phoenix (SLM judges) | Sub-200ms blocking eval. |
| Prompt mgmt | Langfuse (MIT license) | Prompt versioning + tracing. |
| Cost monitoring | OpenObserve + custom dashboards | P99/P50 cost ratio alerting. |
AI-native platforms (LangSmith, Braintrust, Galileo, Maxim) integrate fast but sometimes lock you in. Enterprise APM (Datadog, New Relic) consolidates vendors but often lacks the multi-turn causal analysis complex agents need. If portability > time-to-dashboard: OpenTelemetry + swappable backend.
Implementation checklist (for your next sprint)
- Can you reconstruct the full session trace (retrieval + tools + reasoning) in <5 min when a bug is reported?
- Do you have automated baselines: input distribution, prompt template hash, retrieval corpus overlap?
- Is your instrumentation portable (OTel) or tied to the first SDK you installed?
- Do you capture the P99/P50 cost ratio and alert if it's >50x?
- Do you have a semantic caching layer in front of expensive LLM calls?
If the answer to any of these is "no," that's your first gap. Dashboards, eval rubrics, SLM judges come after. Real observability isn't seeing metrics; it's being able to explain why it failed when everything looks green.