AI Observability — Monitoring LLMs in Production

You’ve deployed an LLM to production. How well is it performing? How much does it cost? Is it hallucinating more? You need AI observability.

What to Measure¶

• Latency: TTFT, total generation time
• Cost: Token usage per request/user/feature
• Quality: User feedback, LLM-as-judge scores
• Errors: API failures, rate limits, timeouts

Tooling¶

LangSmith: Tracing, evaluation. Langfuse: Open-source, self-hostable — our choice. Arize Phoenix: Evals and experiments.

Cost Management¶

• Dashboard with real-time cost per feature
• Alerting on cost anomalies
• Prompt optimization reviews
• Model routing — cheaper model where it suffices

AI Without Observability Is a Ticking Bomb¶

Implement tracing from day one. Langfuse for self-hosted, LangSmith for convenience.