The proof of concept goes well. The cluster gets built, workloads run, everyone is pleased. Then the real work starts and nobody costed it.
The cadence is not optional
Kubernetes releases several times a year and supports a limited number of recent versions. Distributions layer their own lifecycle on top with their own support windows. Fall behind and you are running something unsupported, which means no security fixes and no vendor help during an incident.
The practical consequence: you will be performing a cluster upgrade a few times a year, forever. Not as a project. As routine operations with a named owner and a calendar entry.
Organisations that treat each upgrade as a project end up deferring, and deferred upgrades compound. Skipping versions is usually not supported, so falling three releases behind means three sequential upgrades, each with its own risk.
What actually breaks
Deprecated APIs. Object versions get promoted and old ones eventually removed. Manifests that worked last year fail to apply. This is announced well in advance and still surprises people, because nobody audits their manifests until the upgrade fails.
Run the deprecation checks before every upgrade, not during.
Operators. Each has its own version compatibility matrix against the platform. An upgrade that the platform supports may be blocked by an operator that has not caught up, or worse, may proceed and leave the operator broken.
Inventory your operators with their versions and their upstream health before planning anything. Which leads to the next problem.
Abandoned operators. Somebody installed one from a public catalogue in 2024 to solve a problem. It has not been updated since. Now it blocks your upgrade and you own it.
Node images. The operating system underneath gets updated too, on its own cadence, with its own reboot requirement. On a cluster with strict availability requirements, rolling nodes through maintenance is a scheduling exercise.
Storage drivers. CSI drivers have their own versions and their own compatibility matrix, and storage is exactly where you do not want a surprise.
The disciplines that make it survivable
A non-production cluster that is genuinely representative. Same version, same operators, same storage driver. Upgrade it first, every time. A test cluster that differs from production tests nothing.
Manifests in Git, applied by automation. When everything is declared, rebuilding is possible and drift is visible. When cluster state lives in people’s shell history, you cannot verify anything.
An operator inventory with an owner per entry. Who installed it, why, is it maintained, what happens if it stops working. Review it quarterly. This document prevents more incidents than any monitoring.
Rehearsed rollback. Know what rolling back actually means for your setup, because for some components it means restoring etcd, and that is a different exercise from clicking a button.
The staffing reality
This is the part that gets left out of the business case. A production Kubernetes platform needs continuous attention from people who understand it. Not a fraction of someone’s week between other duties.
Organisations that staff it properly find the platform delivers. Organisations that treat it as something the virtualization team picks up alongside everything else end up with a cluster nobody wants to touch, running an old version, with workloads on it that matter.
That outcome is not a technology failure. It was decided in the business case, by omission, before anything was installed.