Skip to content

Evals & Observability

Production AI fails silently without measurement. This hub ties together evaluation, tracing, monitoring, and safety across the handbook.

The quality loop

flowchart LR
  Build[Build / Prompt] --> Offline[Offline Evals]
  Offline --> Gate{Quality Gate}
  Gate -->|Pass| Deploy[Deploy]
  Gate -->|Fail| Build
  Deploy --> Online[Online Monitoring]
  Online --> Alert[Alerts & Feedback]
  Alert --> Build

Learning path

Step Course Focus
1 Course 13 · LLM Evals Golden sets, LLM-as-judge, CI/CD gates
2 Course 12 · LLMOps Observability, caching, A/B tests
3 Course 08 · Harness L6 Agent step tracing
4 Course 14 · Safety Red teaming, adversarial evals
5 Course 06 · RAG L8 RAG-specific metrics

Eval types

Type When Handbook
Unit / regression Every PR Course 13
RAG retrieval Chunking / index changes Course 06 · RAG eval metrics
Agent trajectory Tool-use agents Course 13 · Agent trajectory evals
Safety / red team Pre-release Course 14 · Red teaming
Online / drift Production Course 13 · Production monitoring

Observability pillars

Pillar What to capture Tools (OSS)
Traces Full request path, agent steps Langfuse, OpenTelemetry, LangSmith
Logs Prompts, tool I/O (redacted) structlog, CloudWatch
Metrics Latency, tokens, cost, error rate Prometheus, Grafana
Feedback Thumbs, edits, escalations Custom + Course 13 monitoring

OSS hubs