Kubernetes in production: 9 mistakes and how to avoid them

A Kubernetes cluster can run fine in testing and fail in production for reasons that never show up in a demo: a node that runs out of memory, an upgrade that takes down every replica at once, or a backup nobody ever restored. The root cause is almost always a default value that nobody reviewed.

This guide walks through nine common mistakes. For each one it explains what happens, how to detect it with kubectl or a manifest, and how to fix it. It relies on the official Kubernetes documentation, the NSA and CISA hardening guidance (version 1.2, August 2022), and the AWS, Google Cloud and Microsoft documentation for their managed services.

The examples apply to the versions supported as of November 4, 2026: 1.35, 1.36 and 1.37. Version 1.34 reached end of life on October 27, 2026 (kubernetes.io/releases).

1. Pods without requests or limits

What happens. The scheduler places each Pod based on its requests, not on its actual usage. If a Pod declares no CPU or memory requests or limits, it lands in the BestEffort quality of service (QoS) class. When a node runs out of resources, Kubernetes evicts BestEffort Pods first, then Burstable and finally Guaranteed. Limits also behave differently: the memory limit is enforced by killing the container (OOM kill) and the CPU limit by slowing it down (throttling). The container does not crash, but it gets slow.

How to detect it.

# Pods without requests or limits (BestEffort QoS)
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass' | grep BestEffort

# Containers terminated for exceeding their memory limit
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilled

How to fix it. Declare requests on every container, based on measured usage, plus a memory limit. If requests and limits are equal for CPU and memory in every container, the Pod is Guaranteed and is last in line for eviction. To keep anything from entering a namespace without values, use a LimitRange, which assigns defaults, and a ResourceQuota to cap the namespace's total usage, as the NSA and CISA guidance recommends.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-values
  namespace: app
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    default:
      memory: 256Mi

2. Misconfigured probes

What happens. The three probes serve different purposes. Liveness restarts the container when it fails; readiness removes it from Services without restarting it; startup holds off the other two until the application has finished starting. The documentation warns that a poorly implemented liveness probe can lead to cascading failures: restarts under load, failed requests and more load on the remaining Pods. The typical case is a liveness probe that queries the database: if the database goes down, every replica restarts at once. Another is a slow application without a startup probe, which gets killed before it finishes starting.

How to detect it.

# Probes that are failing
kubectl get events -A --field-selector reason=Unhealthy

# Pods sorted by restart count
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

How to fix it. Liveness should check only the process itself; external dependencies belong in readiness. Add a startup probe on the same endpoint, with failureThreshold × periodSeconds long enough for the worst-case startup. Keep the defaults in mind: periodSeconds 10, timeoutSeconds 1 and failureThreshold 3.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30   # up to 300 s to start
  periodSeconds: 10
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

3. Images tagged latest or without a digest

What happens. If you don't specify a tag, Kubernetes assumes latest. A tag can be moved to a different image; a digest is a hash of the content and never changes. The official documentation says to avoid :latest in production because it makes it hard to track which version is running and to roll back. With tags that move, you can end up with Pods running different code under the same name. One detail: imagePullPolicy is set when the object is created and is not updated if you change the tag later.

How to detect it.

# Images in use that are not pinned by digest
kubectl get pods -A -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' | sort -u | grep -v '@sha256:'

How to fix it. Use a version tag and add the digest. When both are present, Kubernetes pulls by digest. Scan images in the build pipeline, as the NSA and CISA guidance recommends.

image: registry.example.com/shop/api:1.8.3@sha256:<digest>

4. Exposed Secrets and overly broad RBAC

What happens. By default, Secrets are stored unencrypted in etcd, base64-encoded, which is not encryption. Anyone with API access and read permission can see them. According to the RBAC good practices, list and watch permissions also reveal their contents, and anyone who can create Pods in a namespace can read its Secrets. That is why broad RBAC and unencrypted Secrets are the same problem.

How to detect it.

# Who has cluster-admin
kubectl get clusterrolebindings -o jsonpath='{range .items[?(@.roleRef.name=="cluster-admin")]}{.metadata.name}{"\t"}{.subjects[*].name}{"\n"}{end}'

# What a namespace's default service account can do
kubectl auth can-i --list --as=system:serviceaccount:app:default -n app

# Encryption at rest, on a kubeadm control plane node
sudo grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml

Also check that there are no passwords in ConfigMaps or in manifests stored in Git.

How to fix it. Enable encryption at rest: if kube-apiserver runs without --encryption-provider-config, or the first provider in that file is identity, data is not encrypted. On a managed service, look for the equivalent option in the provider's documentation. Grant permissions with namespaced RoleBindings instead of ClusterRoleBindings, avoid wildcards and don't add users to the system:masters group. Set automountServiceAccountToken: false on Pods that don't use the API.

5. No NetworkPolicies

What happens. By default, a Pod is non-isolated: it accepts all inbound traffic and can open any outbound connection. A compromised Pod can talk to every other Pod. In addition, NetworkPolicies are enforced by the network plugin; if yours doesn't support them, creating them has no effect.

How to detect it. Run kubectl get networkpolicy -A: every namespace missing from that list has open traffic. Confirm in your network plugin's documentation that it supports NetworkPolicy.

How to fix it. The NSA and CISA guidance recommends a default policy that denies all ingress and egress traffic, and then opening only what is needed. Note: denying egress also blocks DNS, so you have to allow it separately.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: app
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: app
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

The k8s-app: kube-dns label is the one used in the DNS documentation; confirm it in your cluster.

6. A single replica and no PodDisruptionBudget

What happens. Draining a node to upgrade it is a voluntary disruption. With a single replica and no PodDisruptionBudget (PDB), a drain leaves the application with no Pods. The opposite mistake also exists: a PDB with maxUnavailable: 0 keeps the drain from ever completing. Managed services handle this in their own way. On EKS, if Pods don't leave the node within 15 minutes, the upgrade fails with PodEvictionFailure unless it is forced. On GKE, in surge upgrades, the drain respects the PDB for up to one hour and then evicts forcibly. On AKS, the default drain timeout is 30 minutes and the pre-upgrade validation checks for misconfigured PDBs.

How to detect it.

# Deployments with a single replica
kubectl get deploy -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas' | awk '$3 == 1'

# Existing PDBs; ALLOWED DISRUPTIONS at 0 blocks drains
kubectl get pdb -A

How to fix it. Run at least two replicas, spread across nodes or zones with anti-affinity rules, and a PDB with maxUnavailable: 1. The documentation prefers maxUnavailable because it adjusts automatically when the number of replicas changes.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: app
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

7. An etcd backup that was never restored

What happens. All Kubernetes objects live in etcd. The snapshot holds the entire cluster state, Secrets included, so it must be encrypted. If a majority of etcd members fail permanently, Pods may keep running, but no new Pods can be scheduled. A backup that has never been restored is not tested.

How to detect and fix it on a self-managed cluster. Check that a recent snapshot exists and do a test restore on another machine. The certificate paths appear in the description of the etcd Pod.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=<ca> --cert=<cert> --key=<key> \
  snapshot save /backups/etcd.db
etcdutl --write-out=table snapshot status /backups/etcd.db
etcdutl --data-dir /tmp/etcd-test snapshot restore /backups/etcd.db

For a real restore, the documentation says to stop all kube-apiserver instances, restore every etcd instance and then start them again. Run etcd with an odd number of members; five are recommended for production.

On a managed service. The provider operates etcd: EKS, for example, places three instances across three Availability Zones. What you need to back up are your resources and volumes, with AWS Backup for EKS, Backup for GKE or an equivalent tool. Backup for GKE, for example, does not back up node pool configuration or networking. Test the restore on a separate cluster.

8. Outdated or out-of-order upgrades

What happens. The project maintains the three most recent minor versions, and each one receives patches for roughly 14 months: 12 of standard support and 2 of maintenance. After that, there are no more security patches. The version skew policy sets rules that are easy to break: kube-apiserver cannot skip minor versions; the kubelet must never be newer than kube-apiserver and may be up to three minor versions older; kubectl is supported within one minor version, older or newer.

How to detect it.

kubectl version
kubectl get nodes   # the VERSION column is the kubelet's

# Deprecated APIs that are still receiving requests
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

How to fix it. Upgrade one minor version at a time, in this order: control plane, nodes (draining them before upgrading the kubelet), clients such as kubectl and, finally, manifests adjusted for API changes. Beforehand, review the deprecated API migration guide; kubectl convert rewrites a manifest to the new version. With kubeadm, kubeadm upgrade plan shows which versions you can upgrade to.

9. No minimum observability

What happens. Kubernetes does not store logs at the cluster level: you need a separate backend. kubectl logs only shows the current file after rotation; if a Pod wrote 40 MiB and the kubelet rotates every 10 MiB, you will see at most 10 MiB. kubectl top requires metrics-server or another adapter. And on a self-managed cluster, if kube-apiserver is not given --audit-policy-file, no audit events are logged.

How to detect it.

kubectl get apiservices | grep metrics.k8s.io
kubectl top nodes
kubectl get events -A --field-selector type=Warning

How to fix it. A reasonable minimum: node and Pod metrics (Kubernetes components expose them in Prometheus format at /metrics), logs shipped outside the cluster, and auditing enabled. The NSA and CISA guidance suggests logging requests involving Secrets only at the Metadata level, so their contents are not copied into the logs. At a minimum, set alerts for repeated restarts, OOMKilled, NotReady nodes and PDBs with zero allowed disruptions.

Pre-production checklist

Sources

• Kubernetes · Releases and Patch Releases.
• Kubernetes · Version Skew Policy.
• Kubernetes · Pod Quality of Service Classes.
• Kubernetes · Resource Management for Pods and Containers.
• Kubernetes · Limit Ranges.
• Kubernetes · Liveness, Readiness, and Startup Probes and Configure Liveness, Readiness and Startup Probes.
• Kubernetes · Images.
• Kubernetes · Secrets and Encrypting Confidential Data at Rest.
• Kubernetes · Role Based Access Control Good Practices and kubectl auth can-i.
• Kubernetes · Network Policies and Debugging DNS Resolution.
• Kubernetes · Disruptions and Specifying a Disruption Budget.
• Kubernetes · Operating etcd clusters for Kubernetes.
• Kubernetes · Upgrade A Cluster, Deprecated API Migration Guide and Deprecation Policy.
• Kubernetes · Logging Architecture, Resource metrics pipeline, Metrics For Kubernetes System Components and Auditing.
• NSA and CISA · Kubernetes Hardening Guidance, version 1.2 (August 2022).
• AWS · Amazon EKS architecture, Managed node update behavior and Back up your EKS Clusters with AWS Backup.
• Google Cloud · GKE node pool upgrade strategies and Backup for GKE.
• Microsoft · AKS upgrade options.

If you want to go through this checklist with your team before going to production, let's talk.