Taking over a Kubernetes environment somebody else built is a routine part of this work. The cluster runs, mostly. There is nobody left who can explain all of it. Something is wrong often enough to be annoying but not often enough to have been fixed.
The instinct is to start fixing. Resist it for a day. The order below front-loads the questions whose answers change everything downstream.
1. Establish what is actually running
Before opinions, inventory. You want to know what exists and how it got there.
- Cluster version, and the version skew between control plane and nodes.
- Managed add-on versions — VPC CNI, CoreDNS, kube-proxy, EBS CSI driver — and how far behind they are.
- What is deployed in every namespace, including the ones that look abandoned.
- Where each workload's manifests live, and whether the cluster matches them.
That last point is the one that matters most. If the running state does not match any repository, you are not operating a declarative system — you are operating a mutable one that happens to speak YAML, and every later assumption changes accordingly.
2. Find out how changes reach the cluster
Ask: if I need to change a deployment, what is the correct path? The answer falls into one of a few categories, and each implies different work.
- GitOps — a controller reconciles from a repository. Good. Verify it is actually reconciling and not stuck or suspended, and check whether anyone has been bypassing it with direct
kubectl apply. - Pipeline-driven — CI applies manifests or Helm charts. Workable. Find out who can trigger it and what happens on failure.
- Manual — someone applies from a laptop. This is the finding that reorders your whole plan, because until it changes, nothing else you fix will stay fixed.
3. Check the upgrade position
EKS versions have support windows, and the cost of an upgrade rises sharply once you are several versions behind — not because the upgrade itself is harder, but because deprecated APIs accumulate and add-on compatibility narrows.
Establish: how many versions behind you are, when standard support for the current version ends, and whether any workload uses an API removed in a version you need to pass through. That last one is worth checking with tooling rather than by reading manifests, because the failure mode is a resource silently failing to apply after the upgrade.
An out-of-support cluster is usually the first thing to schedule, ahead of nicer improvements, because everything else you build sits on top of it.
4. Look at resource requests and limits honestly
In most inherited clusters, requests were guessed once during initial deployment and never revisited. The consequences run in both directions.
- Requests far above real usage — the cluster is paying for capacity nothing uses, and the scheduler cannot pack nodes efficiently.
- Requests far below real usage — nodes are oversubscribed, and pods are evicted or throttled under load in ways that look like application bugs.
- No limits on memory — one leaking pod can take a node's other workloads with it.
- Aggressive CPU limits — throttling that presents as latency, and is invisible unless you are looking at throttling metrics specifically.
Compare requests against several weeks of actual usage before changing anything. The point is not to optimize immediately; it is to know which workloads are configured on the basis of evidence and which are configured on the basis of a guess someone made in a hurry.
5. Trace what happens when a node disappears
Nodes are replaced routinely — scaling events, instance refreshes, spot interruption, upgrades. Whether that is a non-event or an outage depends on configuration most teams never test.
Check that workloads that need it have more than one replica, that pod disruption budgets exist and are not so strict that they block drains entirely, that anti-affinity actually spreads replicas across nodes and zones, and that termination grace periods are long enough for in-flight requests to finish.
Then verify it rather than reasoning about it. Drain a node during working hours and watch what happens. Doing that deliberately, once, while you are paying attention, is worth more than any amount of configuration review — and it is much better than discovering the answer during an unplanned interruption.
6. Establish whether you would know about a problem
The observability question is not "is monitoring installed." It is: if a workload started failing right now, how would we find out, and how long would it take to identify which one?
Concretely: are pod restarts, OOM kills, failed scheduling and node pressure visible somewhere an engineer looks? Are logs retained long enough to investigate something that happened last week? Do alerts point to an action, or are they noise people have learned to dismiss?
An alert everyone ignores is worse than no alert, because it creates the impression of coverage.
7. Review access before you widen it
Inherited clusters accumulate access. Look at who and what can reach the API server, whether the endpoint is public, which IAM principals map to cluster-admin, and whether workload service accounts have narrowly scoped IAM roles or share something broad.
Service account permissions are the common finding. A pod that only needs to read one bucket frequently has credentials that can read everything — usually because a role was created once, worked, and got reused.
What this gets you
At the end of this pass you should be able to answer, in a sentence each: how changes reach the cluster, how far behind it is, whether its resource configuration is evidence-based, whether it survives node replacement, whether you would notice a failure, and who can do what.
Those six answers determine what to fix first. Without them you are making changes to a system whose behaviour you are guessing at — which is how a stable-but-annoying cluster becomes an unstable one.