DevOps Intermediate
Grafana Tempo — Distributed Tracing¶
TempoTracingGrafanaObservability 5 min read
Grafana Tempo for distributed tracing. Object storage backend, TraceQL queries and Grafana stack integration.
What is Tempo¶
Grafana Tempo is a highly scalable tracing backend. It stores traces directly in object storage — no Elasticsearch/Cassandra needed.
- Object storage only — low cost
- TraceQL — query language for traces
- Native integration with Grafana, Loki, Mimir
Deployment¶
tempo:
image: grafana/tempo:2.4.0
command: ["-config.file=/etc/tempo.yaml"]
ports:
- "4317:4317" # OTLP gRPC
- "3200:3200" # Tempo API
# tempo.yaml
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
http:
storage:
trace:
backend: s3
s3:
bucket: tempo-traces
endpoint: s3.eu-west-1.amazonaws.com
TraceQL¶
# Traces with errors
{ resource.service.name = "order-service" && status = error }
# Traces longer than 2 seconds
{ duration > 2s }
# HTTP 500 errors
{ span.http.status_code >= 500 }
# Pipeline
{ resource.service.name = "api-gateway" } | avg(duration) > 500ms
Summary¶
Grafana Tempo offers cost-effective distributed tracing with simple operations. TraceQL and native Grafana integration are unmatched.
Need Help with Implementation?¶
Our team has experience designing and implementing modern architectures. We’re happy to help.