LLM Monitoring v2 — From Logging to Predictive Observability

Logging LLM calls is baseline. In 2025: real-time quality scoring, embedding drift detection, predictive alerting.

Beyond Logging¶

• Real-time quality: Every response scored inline
• Embedding drift: Auto-detect changes in query distribution
• Predictive cost: Forecast AI spending
• User satisfaction: Correlation of feedback vs quality scores

Stack 2025¶

Langfuse for tracing. Arize Phoenix for evaluations. Grafana for business metrics. PagerDuty for alerts.

Alert Fatigue¶

Quality drop >10% sustained 1h → alert. Cost spike >50% → alert. Error rate >5% → immediate. Everything else → daily digest.

Observability Is the New Testing¶

In the non-deterministic LLM world, production monitoring is more important than pre-production testing.