Cluster-level errors

RBAC & Auth

1. RBAC "forbidden" — user or SA lacks permission

kubectl auth can-i list pods --as jane      # test as user
kubectl auth can-i list pods --as system:serviceaccount:dev:ci   # test as SA

Fix by creating a Role/ClusterRole + Binding.

2. Service Account token expired (1.24+)

K8s 1.24 stopped auto-generating secret-based tokens. Use projected token volumes or TokenRequest API.

3. Kubelet certificate expired

kubectl on a node fails with x509: certificate expired.

kubeadm certs check-expiration
kubeadm certs renew all
systemctl restart kubelet

Scaling

4. HorizontalPodAutoscaler not scaling

Metrics server missing or unhealthy.

kubectl top pods                      # if this fails → metrics-server broken
kubectl -n kube-system get pods -l k8s-app=metrics-server
kubectl describe hpa <hpa>            # check "Metrics" section

Also confirm the pods actually set resources.requests (HPA needs them for CPU%).

5. Cluster Autoscaler not adding nodes

• Cloud IAM missing the Autoscaling.* permissions.
• Node group at max size.
• Pod requests bigger than any allowed node type.

6. ResourceQuota exceeded

kubectl -n <ns> describe resourcequota

Either raise the quota or trim pod requests.

7. Pod priority & preemption

priorityClassName set on the pod but low priority class hasn't been created. Or preemption disabled cluster-wide.

Storage

8. PVC stuck in Pending

kubectl describe pvc <pvc>
kubectl get storageclass

• No StorageClass matches the PVC's storageClassName.
• Provisioner (EBS CSI, Azure Disk CSI…) is not running or lacks IAM.

9. Volume plugin not loaded

Kubelet logs: cannot find volume plugin. Install the CSI driver DaemonSet.

10. Volume mount permission denied → see Part 1 (#11).

Scheduling

11. Scheduler not scheduling

kubectl -n kube-system logs -l component=kube-scheduler

• Only 1 scheduler pod? On managed clusters this is rare — could be leader-election flapping.

12. Pod affinity / anti-affinity impossible

Rule requires a node label that no node has, or two anti-affine pods but only 1 node.

13. Taints without matching tolerations

kubectl describe node <node>     # check Taints

Add tolerations to the pod, or kubectl taint node <node> key-.

14. NodeDrain fails

kubectl drain node1 blocked because a PDB refuses eviction.

kubectl get pdb -A

Temporarily loosen the PDB or wait for another replica to become ready.

Control plane

15. API server unreachable

• Load balancer in front of control plane broken.
• kubeconfig points to the wrong server URL.

16. ETCD cluster unhealthy

etcdctl endpoint status --write-out=table

Common causes: disk full on ETCD nodes, high latency between members, corrupted data (restore from snapshot).

17. Inconsistent API versions

kubectl client 3+ minor versions behind server → weird errors. Upgrade kubectl.

18. CRD issues

Custom Resource created but controller not installed → the resource sits in etcd doing nothing.

19. ImageRegistryAuthentication (private repo)

• Create a docker-registry secret and reference it via imagePullSecrets.
• On EKS with ECR → attach AmazonEC2ContainerRegistryReadOnly to the node role.

20. RuntimeClass missing

Pod requests a RuntimeClass: gvisor but the class isn't installed. Install gVisor and the RuntimeClass, or remove the field.

The universal first-step

kubectl get events --sort-by='.lastTimestamp' -A | tail -30

Nine times out of ten, the answer is already in the event stream.

Real-world example

kubectl fails cluster-wide with x509: certificate has expired. You SSH the master, kubeadm certs check-expiration → all client certs expired yesterday. kubeadm certs renew all + systemctl restart kubelet on every control-plane node → cluster alive again in 10 minutes. Now you add a Cron alert 30 days before expiry.