Lesson 24 / 32

Logging, Metrics, Tracing & Alerting

Knowing what your system is doing before users tell you it's broken.

The three pillars

Logs are timestamped discrete events (good for 'what exactly happened here'). Metrics are aggregated numbers over time (good for trends and dashboards — request rate, error rate, latency). Traces follow one request across every service it touched (good for finding where in a call chain the time went).

The golden signals

For any service, watch latency (p50/p95/p99, not just average), traffic (requests/sec), errors (rate of failed requests), and saturation (how full is CPU, memory, queue depth). These four catch most incidents early.

A car dashboard

Metrics are the speedometer and fuel gauge — glance and know your state. Logs are the trip computer's detailed event history. Tracing is like a black box recorder that shows the exact sequence of what each part of the car did leading up to a specific trip.

Alert on symptoms, not causes

Alert on user-facing symptoms (error rate spiked, p99 latency breached SLO) rather than every internal cause (one server's CPU is high). Symptom-based alerts reduce noise and point straight at what users are experiencing.