Kubernetes 1.5 — Finally Ready for Production?

After a year of experimenting with Docker Swarm and Mesos, we decided to bet on Kubernetes. Version 1.5 introduced StatefulSets (beta) and PodDisruptionBudgets — exactly what we needed for our enterprise deployments.

Why Kubernetes and Not Docker Swarm¶

Docker Swarm is simple. Too simple. For a few containers on a single project, it’s great, but once you need RBAC, namespace isolation, a custom scheduler, or federation across multiple clusters, Swarm falls short. And that’s exactly where we found ourselves.

Kubernetes has a steeper learning curve — that’s a fact. But the community around it is massive — Google, Red Hat, CoreOS, and dozens of other companies are investing in it. For us, that’s an important signal of stability. We don’t want to build infrastructure on a project nobody maintains two years from now.

What Version 1.5 Brought¶

StatefulSets (beta). Formerly PetSets — finally a way to run stateful workloads. Databases, message brokers, Elasticsearch clusters. Each pod gets a stable hostname and persistent volume. Critical for us — we have clients who need PostgreSQL in a cluster.

PodDisruptionBudgets. You tell Kubernetes: “at least 2 replicas of my service must always be running.” The cluster then respects this constraint during rolling updates or node drains. Finally, maintenance without downtime.

Cluster Federation. Alpha, but promising. Imagine a single kubectl command that deploys an application to clusters in Prague and Frankfurt. For disaster recovery and latency-based routing, this will be a game changer.

Our Test Topology¶

We spun up a three-node cluster on bare metal servers in our data center. Three masters with etcd in HA mode, three worker nodes. Networking via Calico — we need network policies for tenant environment isolation.

$ kubectl get nodes
NAME      STATUS    AGE
master1   Ready     45d
master2   Ready     45d
master3   Ready     45d
worker1   Ready     44d
worker2   Ready     44d
worker3   Ready     44d

Installation via kubeadm — surprisingly painless. We’d previously tried the manual setup following Kelsey Hightower’s “Kubernetes The Hard Way,” which was educational, but for production we want automation.

First Production Workload¶

As our pilot project, we chose the internal monitoring stack — Prometheus + Grafana. An ideal candidate: stateless (Grafana), semi-stateful (Prometheus with 30-day retention), and if it goes down, nobody loses money.

The deployment went smoothly. The Helm chart for the Prometheus stack worked nearly out of the box; we just had to adjust the storage class for our Ceph volumes. Grafana dashboards were defined as ConfigMaps — infrastructure as code, as it should be.

Where Kubernetes Surprised Us Negatively¶

Debugging. When something doesn’t work, you have to go through kubelet, kube-proxy, kube-controller-manager, and kube-scheduler logs… Troubleshooting tooling is still weak. kubectl describe pod is your best friend, but sometimes it’s not enough.

Resource management. Setting requests and limits correctly is an art. Set limits too low — OOM kill. Set them too high — wasted resources.

Upgrades. The upgrade from 1.4 to 1.5 wasn’t entirely trivial. etcd upgrade, API server flags changed. Zero-downtime upgrade of the cluster itself is still a challenge.

Plan for 2017¶

By the end of Q2, we want at least 5 internal services on Kubernetes. By year-end, we plan to deploy the first client project on a container platform. We’re tracking Kubernetes 1.6 (RBAC as beta!) and the upcoming EKS from Amazon.

Kubernetes Is the Future of Our Infrastructure¶

It’s not a simple tool and it requires an investment in know-how. But for a company our size, managing dozens of services for different clients, container orchestration is a necessity. Kubernetes is the best choice on the market.