SRE — Four Golden Signals in Practice

DevOps Intermediate

SRE — Four Golden Signals in Practice¶

SREGolden SignalsMonitoringObservability 5 min read

Implementing the Four Golden Signals: Latency, Traffic, Errors, Saturation with PromQL queries and alerting.

The Four Golden Signals¶

• Latency — request processing time
• Traffic — volume of requests (RPS)
• Errors — percentage of failed requests
• Saturation — how full the system is (CPU, RAM, disk, connections)

PromQL Implementation¶

# Latency (p99)
histogram_quantile(0.99, sum by (le) (
  rate(http_request_duration_seconds_bucket[5m])))

# Traffic
sum(rate(http_requests_total[5m]))

# Errors
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100

# Saturation
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

Alerting Rules¶

groups:
  - name: golden-signals
    rules:
      - alert: HighLatency
        expr: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 5m
        labels:
          severity: warning
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.01
        for: 2m
        labels:
          severity: critical
      - alert: HighSaturation
        expr: 1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.85
        for: 10m
        labels:
          severity: warning

Summary¶

The Four Golden Signals are the simplest framework for service monitoring. They cover 90% of needs with a minimum of metrics.

Need Help with Implementation?¶

Our team has experience designing and implementing modern architectures. We’re happy to help.

Free Consultation