Every object in a Kubernetes cluster — every pod, every secret, every configuration — lives in etcd. The API server is a front door onto it. Controllers watch it. When etcd is unhealthy, everything above behaves strangely in ways that look like application problems.
It is also the thing least likely to be monitored in a cluster built by people who came from elsewhere.
What etcd needs
Low-latency storage. It writes to a persistent log and fsyncs. Write latency directly determines cluster responsiveness. The guidance is dedicated fast disks, and dedicated means not sharing spindles or queues with workload storage.
This is the single most common cause of mysterious cluster slowness in environments where control plane nodes were placed on general-purpose shared storage because that was what was available.
Low-latency networking between members. It runs a consensus protocol. Members vote. Network latency between them is in the critical path of every write, and a stretched cluster across sites with meaningful latency will behave badly.
An odd number of members. Three or five. Quorum is a majority, so four members give you no more fault tolerance than three while adding coordination cost.
The symptoms of an unhappy etcd
Slow kubectl responses across the board. Leader elections in the logs. Controllers reconciling sluggishly. Pods taking a long time to schedule. Intermittent API timeouts under load.
These present as “Kubernetes is slow”, which sends people to look at the workloads, where they find nothing.
The metrics that matter
Disk sync duration for the write-ahead log and for backend commits. Leader changes over time, which should be approximately zero in a healthy cluster. Database size against the quota. Proposal failures. Round-trip time between members.
Put these on a dashboard on day one. If you build one dashboard for your platform, build this one. It is the difference between diagnosing a problem and guessing at it.
The database size trap
etcd has a storage quota. Exceed it and the cluster goes read-only, which is an outage that arrives without much warning.
Size grows with object count and, crucially, with revision history. Compaction and defragmentation are maintenance operations that must happen. Managed distributions usually handle this automatically; verify that yours does rather than assuming.
The things that fill it unexpectedly: excessive secrets, controllers that update objects in a tight loop, custom resources being written at high frequency by something nobody is watching. An operator with a bug can fill etcd faster than you would believe.
Backups
An etcd snapshot is a complete backup of cluster state. Taking one is a single command and takes seconds.
Restoring it is not a single command, and the procedure differs by distribution. Read your distribution’s restore procedure now, and rehearse it on a test cluster, because the day you need it will be a bad day and the documentation is not written for people who are panicking.
Also: an etcd snapshot contains your secrets. Protect it accordingly — encrypted at rest, access controlled, and not sitting on a share that half the organisation can read.
The habit
When something is wrong with the cluster and the workloads look fine, check etcd first. Disk latency, leader elections, database size. In my experience that is where the answer is often enough that it should be the first place you look rather than the last.