DevOps Intermediate
Grafana Loki — Log Aggregation¶
LokiLoggingGrafanaObservability 6 min read
Grafana Loki for centralized log collection. Architecture, LogQL queries, label design and integration.
Why Loki¶
Loki is “Prometheus for logs” — it indexes only labels, not log content. Orders of magnitude lower costs than Elasticsearch.
- Label-based indexing
- Compatible with Prometheus labels
- Object storage backend
- Native integration with Grafana
Promtail Agent¶
server:
http_listen_port: 9080
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
pipeline_stages:
- json:
expressions:
level: level
- labels:
level:
LogQL¶
# Basic filter
{namespace="production", app="api-server"} |= "error"
# JSON parsing + filter
{app="api-server"} | json | level="error" | status >= 500
# Metrics from logs
sum(rate({app="api-server"} |= "error" [5m])) by (namespace)
# Top 10 most frequent errors
topk(10, sum by (error_type) (
count_over_time({app="api-server"} | json | level="error" [1h])))
Label Design¶
- Low cardinality:
namespace,app,environment,level - Avoid high cardinality:
user_id,request_id,IP - Rule: max 10-15 unique label combinations per stream
Summary¶
Loki offers cost-effective log aggregation thanks to label-based indexing. LogQL and native Grafana integration enable correlation of logs with metrics and traces.
Need Help with Implementation?¶
Our team has experience designing and implementing modern architectures. We’re happy to help.