Kubernetes in First Production — What Three Months of Operations Taught Us

Remember when I wrote that Kubernetes is “overkill for 3 services”? We now have 12 services and Kubernetes is exactly what we need. After three months of production operations: it’s a powerful tool that can hurt you.

Why Kubernetes and Not Docker Swarm¶

Swarm is simpler. But the community is behind Kubernetes — Google, Red Hat, CoreOS, Microsoft. The ecosystem (Helm charts, operators, monitoring) is an order of magnitude larger. K8s handles rolling updates with health checks, ConfigMaps, Secrets, RBAC, Ingress.

Cluster Setup — kubeadm on Bare Metal¶

The client wants data in the Czech Republic on their own hardware. Three master nodes (HA etcd), five worker nodes, RHEL 7. Kubeadm, Calico networking, MetalLB. Setup took two weeks — the learning curve is steep.

Persistent Storage — The Toughest Nut to Crack¶

Stateless services on k8s are a breeze. Stateful? A nightmare. We tried NFS (unstable), Ceph (complex), and finally GlusterFS. Lesson learned: databases outside Kubernetes. PostgreSQL on a dedicated server, services connect via ExternalName service.

Deployment Workflow¶

Jenkins → Docker image → Harbor registry → Helm upgrade → rolling update with readiness/liveness probes = zero-downtime deployment.

replicaCount: 3
image:
  repository: registry.core.internal/myapp
  tag: "1.23.4"
resources:
  requests: { memory: "512Mi", cpu: "250m" }
  limits: { memory: "1Gi", cpu: "500m" }
readinessProbe:
  httpGet: { path: /health, port: 8080 }
  initialDelaySeconds: 30

Monitoring — Prometheus + Grafana¶

Prometheus scrapes metrics, Grafana visualizes, Alertmanager sends to Slack. Kube-state-metrics, cAdvisor, node-exporter — three layers of monitoring.

What Bit Us¶

Resource limits — a pod consumed an entire node without limits. Pod scheduling — two replicas on the same node = false HA. etcd backup — without it, you don’t have a cluster. YAML hell — hundreds of lines for 12 services.

Kubernetes Isn’t for Everyone¶

For a small company with 3 services — Docker Compose or managed PaaS. For 10+ microservices — Kubernetes saves operational pain. Just expect an upfront investment.