41. Autoscaling during a dependency outage
Workers autoscale on queue depth. When a downstream API went down, every job failed and was requeued, so the queue grew and workers scaled from 10 to 200, hammering the API as it tried to recover. What fixes this?
- A.Scale on CPU instead, since it stays the same
- B.Raise the maximum to 500 workers
- C.Delete the queue when it passes 10,000 jobs
- D.Back off or pause on dependency errors and cap scale
Show answer
Answer: D. Back off or pause on dependency errors and cap scale
Queue depth rose because of failures, not new demand, so scaling added load to the broken dependency. Workers should back off (or trip a breaker and pause) on dependency errors, and autoscaling should have a sane maximum or factor in error rate. Scaling only helps when the bottleneck is the workers themselves.
42. Container ignores SIGTERM
Pods with this image always take the full 30-second grace period to stop and are then SIGKILLed, dropping requests. What is the most likely cause?
FROM python:3.12-slim
COPY app.py .
CMD python app.py
- A.Shell form CMD runs /bin/sh -c as PID 1, which does not forward SIGTERM to the Python process; use exec form CMD ["python", "app.py"] and handle SIGTERM in the app.
- B.The python:3.12-slim base image blocks all signals by default.
- C.terminationGracePeriodSeconds must be set to 0 for signals to be delivered.
- D.Kubernetes sends SIGINT, not SIGTERM, so Python ignores it.
Show answer
Answer: A. Shell form CMD runs /bin/sh -c as PID 1, which does not forward SIGTERM to the Python process; use exec form CMD ["python", "app.py"] and handle SIGTERM in the app.
With shell form, the shell becomes PID 1 and Python is its child. The shell usually does not pass SIGTERM on, and PID 1 has no default signal handlers, so nothing stops until the kubelet sends SIGKILL. Exec form makes Python PID 1 directly; the app should still handle SIGTERM, or you can use an init such as tini.
43. Rolling update that never finishes
A Deployment uses maxUnavailable: 0 and maxSurge: 1. The cluster has no spare capacity and no autoscaler. What happens during a rollout?
- A.Kubernetes deletes an old Pod first to make room, then starts the new one.
- B.The rollout finishes instantly because maxSurge is ignored when the cluster is full.
- C.Kubernetes evicts Pods from other Deployments to make space.
- D.The new surge Pod stays Pending, no old Pod is removed, and the rollout stalls until progressDeadlineSeconds marks it as failed.
Show answer
Answer: D. The new surge Pod stays Pending, no old Pod is removed, and the rollout stalls until progressDeadlineSeconds marks it as failed.
With maxUnavailable: 0, the controller must keep the full replica count available, so it can only remove an old Pod after a new one is Ready. With no room, the surge Pod cannot be scheduled, so nothing progresses. Without PriorityClasses that allow preemption, other workloads are not evicted. Fixes include adding capacity, an autoscaler, or allowing maxUnavailable: 1.
44. Errors during every rollout
Your app handles SIGTERM correctly and stops accepting connections right away. Still, clients see a few 502 errors at every rollout. What is the most likely cause and fix?
- A.The readiness probe period is too long; set it to 1 second and the errors disappear.
- B.Endpoint removal reaches load balancers and kube-proxy asynchronously, so traffic still arrives after SIGTERM; add a short preStop sleep (or delay shutdown) so the Pod keeps serving until routing is updated.
- C.Kubernetes sends SIGKILL before SIGTERM; increase terminationGracePeriodSeconds to 300.
- D.The Service must be type LoadBalancer for graceful shutdown to work.
Show answer
Answer: B. Endpoint removal reaches load balancers and kube-proxy asynchronously, so traffic still arrives after SIGTERM; add a short preStop sleep (or delay shutdown) so the Pod keeps serving until routing is updated.
When a Pod is deleted, the kubelet sends SIGTERM while, in parallel, the endpoint is removed and that change spreads to kube-proxy, ingress controllers and cloud load balancers. For a few seconds traffic can still arrive at a Pod that has already stopped listening. A preStop sleep of a few seconds, plus continuing to serve in-flight requests, closes this gap.
45. Docker cache on fresh runners
You moved to ephemeral CI runners, and Docker builds that took 2 minutes now take 12 because every layer is rebuilt. How do you keep layer caching?
- A.Use docker build --no-cache so builds stay consistent
- B.Turn the ephemeral runners back into long-lived ones
- C.Export the BuildKit cache to a registry with --cache-to and reuse it
- D.Squash the image into one layer before pushing
Show answer
Answer: C. Export the BuildKit cache to a registry with --cache-to and reuse it
Each fresh runner starts with an empty local cache. BuildKit can push cache to a registry or the CI cache service (--cache-to type=registry,ref=..., or type=gha) and load it with --cache-from on the next build. You keep clean runners and get fast builds back.
46. Trusted registries and digests only
You want production Pods to run only images from your registry, pinned by digest, so a tag cannot be repointed after review. What enforces this for every deploy?
- A.A Dockerfile lint rule that checks each FROM line
- B.An admission policy allowing only your registry and digests
- C.A firewall rule that blocks Docker Hub on office laptops
- D.A CI step that prints the image name before each deploy
Show answer
Answer: B. An admission policy allowing only your registry and digests
An admission policy (Kyverno, Gatekeeper or ValidatingAdmissionPolicy) can reject any Pod whose image is outside the approved registry or lacks an @sha256 digest; Kyverno can also resolve tags to digests. It applies to every path into the cluster. Lint rules and laptop firewalls cover only part of the path.
47. Pod create rights reveal Secrets
A developer has no get or list rights on Secrets in namespace payments, but can create Pods there. Why can they still read the payments DB password Secret?
- A.They can make a Pod that mounts the Secret and prints it
- B.Pods inherit cluster-admin rights by default
- C.Secrets are shown in kubectl get pods output
- D.The scheduler sends Secret values to the Pod creator
Show answer
Answer: A. They can make a Pod that mounts the Secret and prints it
RBAC checks the creator's right to create Pods, not to read the Secrets those Pods mount, so a Pod can mount any Secret in its namespace. Pod create rights in a namespace effectively include reading its Secrets. Separate sensitive workloads into their own namespaces and limit who can create workloads there.
48. Namespace stuck in Terminating
You deleted a namespace an hour ago and it still shows Terminating. Its status says some resources have finalizers remaining, and a CRD's operator was uninstalled before the namespace was deleted. What is happening?
- A.The namespace is waiting for its ResourceQuota to reset
- B.Kubernetes deletes namespaces only once per day
- C.The API server is still downloading the namespace backup
- D.Objects wait on finalizers that no controller is left to clear
Show answer
Answer: D. Objects wait on finalizers that no controller is left to clear
A finalizer tells Kubernetes to wait for a controller to finish cleanup before deleting an object. If the operator is gone, nothing removes the finalizer, so those objects and the namespace never finish. Reinstall the operator to clean up, or, knowing the risk of leaked external resources, remove the finalizers by hand.
49. Changing a Deployment selector
You change `spec.selector.matchLabels` from app: web to app: web-frontend in a Deployment and apply it. The API server rejects it with "field is immutable". What is the safe way forward?
- A.Patch the selector with kubectl edit, which skips the check
- B.Delete the ReplicaSets so the selector can be changed
- C.Add --force-conflicts to kubectl apply
- D.Create a new Deployment with the new selector, then remove the old one
Show answer
Answer: D. Create a new Deployment with the new selector, then remove the old one
In apps/v1 the Deployment selector cannot change, because it decides which ReplicaSets and Pods the Deployment owns. To move labels, create a new Deployment (and adjust the Service selector), let it become ready, then delete the old one. Deleting and recreating the same name works too but causes downtime.
50. Restart delay in CrashLoopBackOff
A container crashes right after starting. The kubelet restarts it with a growing delay. Under the default settings, what happens to the delay as the crashes continue?
- A.It stays fixed at 30 seconds between every restart
- B.It grows without limit until the Pod is deleted
- C.It doubles from about 10 seconds and is capped at 5 minutes
- D.It resets to zero each time the node runs low on memory
Show answer
Answer: C. It doubles from about 10 seconds and is capped at 5 minutes
The kubelet uses exponential back-off: about 10s, 20s, 40s and so on, up to a 5-minute cap. The back-off resets after a container runs successfully for 10 minutes. Long waits between attempts are normal, so check kubectl logs --previous rather than waiting for the next try.
51. ConfigMap file mounted with subPath
A ConfigMap key is mounted at /etc/nginx/conf.d/app.conf using subPath, so the other files in that folder stay. After you edit the ConfigMap, the file inside the running Pod never changes. Why?
- A.subPath mounts copy the file once and do not receive updates
- B.ConfigMaps mounted as files never update in any form
- C.nginx locks the file so the kubelet cannot replace it
- D.Edits apply only to ConfigMaps in the kube-system namespace
Show answer
Answer: A. subPath mounts copy the file once and do not receive updates
Normal ConfigMap volumes are updated by swapping a symlink in the mounted folder. A subPath mount binds one file directly, so it misses those swaps and stays at the old content until the Pod restarts. Mount the whole ConfigMap in its own folder, or roll the Pods on config changes.
52. Spotting a Guaranteed QoS Pod
Each Pod below has one container. Which Pod gets the Guaranteed QoS class?
- A.requests cpu 500m / memory 512Mi, limits cpu 1 / memory 512Mi
- B.limits cpu 500m and memory 512Mi, with no requests set
- C.requests and limits memory 512Mi, with no CPU values set
- D.no requests or limits, but a high PriorityClass
Show answer
Answer: B. limits cpu 500m and memory 512Mi, with no requests set
Guaranteed needs CPU and memory limits on every container with requests equal to limits. If you set only limits, Kubernetes copies them into requests, so that Pod qualifies. The second Pod has different CPU values and the third lacks CPU, so both are Burstable; the last is BestEffort, as priority does not affect QoS.
53. Where the next Pod can go
A Deployment uses topologySpreadConstraints on zone with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule. Current matching Pods: zone-a 4, zone-b 2, zone-c 1. Which zone can take the next Pod?
- A.zone-b or zone-c
- B.Only zone-c
- C.Any of the three zones
- D.None of them, so it stays Pending
Show answer
Answer: B. Only zone-c
For each candidate zone the skew is (Pods there + 1) − (lowest count among zones). zone-c gives 2 − 1 = 1, which is allowed; zone-b gives 3 − 1 = 2 and zone-a gives 5 − 1 = 4, both above maxSkew. The constraint pushes new Pods toward the emptiest zone, even when the current spread is already uneven.
54. EKS pods cannot get IP addresses
On EKS with the default VPC CNI, new pods stay in ContainerCreating with "failed to assign an IP address to container". The node subnets are small /24s and nearly full. What is the best fix?
- A.Use larger node instance types so each node can host more pods
- B.Change every Service to type LoadBalancer so pods need no IPs
- C.Delete the CoreDNS pods to free up addresses in the subnets
- D.Add pod address space via a secondary CIDR or prefix delegation
Show answer
Answer: D. Add pod address space via a secondary CIDR or prefix delegation
The AWS VPC CNI gives every pod a real VPC IP from the node subnet, so small subnets run out fast. Adding a secondary CIDR (for example from 100.64.0.0/10) with custom networking, or enabling prefix delegation with enough free contiguous space, solves it. Bigger nodes allow more pods per node but use up subnet IPs even faster.
55. Slow pods at 30% CPU
Pods on a managed Kubernetes cluster show only 30% average CPU, but latency spikes. Metrics show heavy CFS throttling. What is the likely cause?
- A.Requests are too high, so pods cannot schedule
- B.The node pool uses the wrong region
- C.CPU limits are too low, so bursts are throttled
- D.Memory limits throttle the CPU
Show answer
Answer: C. CPU limits are too low, so bursts are throttled
CPU limits are enforced in short periods; a bursty app hits its limit within a period and is paused, even if average use is low. Raise or remove CPU limits while keeping sensible requests for scheduling.
56. Throttled but not busy
A multithreaded service in a Kubernetes pod with a CPU limit of 1 core shows only 40% average CPU usage, but p99 latency is high and the container_cpu_cfs_throttled_periods_total metric keeps rising. What is the most likely cause?
- A.The node is out of memory and swapping
- B.The readiness probe is failing
- C.The CPU request is too high, so the scheduler gives it less CPU
- D.The CFS quota is enforced per 100 ms period; bursts across many threads use up the quota early in the period and the container is paused until the next period
Show answer
Answer: D. The CFS quota is enforced per 100 ms period; bursts across many threads use up the quota early in the period and the container is paused until the next period
CPU limits are enforced with the CFS quota in short periods (100 ms by default). Many threads can burn the whole quota in a few milliseconds, then wait, which adds latency even though the average looks low. Fixes include raising or removing the limit, matching thread counts to the limit, or tuning the runtime.
57. Which pods are evicted first?
A node is under memory pressure and the kubelet starts evicting pods. All pods use more memory than they request except the Guaranteed ones. Which are evicted first?
- A.BestEffort pods with no requests or limits
- B.Guaranteed pods because they use the most memory
- C.The pods that were scheduled most recently
- D.Pods in the kube-system namespace
Show answer
Answer: A. BestEffort pods with no requests or limits
Under node pressure the kubelet ranks pods by whether usage exceeds requests, then by priority, then by usage relative to requests. BestEffort pods have no requests, so any usage exceeds them and they go first, followed by Burstable pods over their requests. Guaranteed pods within their limits are the last to be evicted.
58. Many DNS queries per lookup in pods
In a Kubernetes pod, resolv.conf has "options ndots:5" and three search domains. Calls to api.partner.com are slow, and DNS logs show several NXDOMAIN queries before the real one. Why?
- A.ndots:5 forces five retries for every query
- B.The pod sends each query to five different servers
- C.Names with under 5 dots try the search domains first
- D.ndots:5 turns off caching for external names
Show answer
Answer: C. Names with under 5 dots try the search domains first
With ndots:5, a name with fewer than five dots is treated as relative, so the resolver tries each search suffix before the name as given. A trailing dot (api.partner.com.) or a lower ndots in the pod dnsConfig avoids the extra lookups.
59. Old run cannot be rebuilt
You must reproduce a model trained 8 months ago. Its Dockerfile used "FROM python:3.10-slim", and that build of the base image is no longer the same. What should the team have done?
- A.Used the "latest" tag so it was always kept current
- B.Rebuilt the image weekly from the very same Dockerfile
- C.Kept only the Dockerfile in Git and never the image
- D.Pinned the base image digest and mirrored it internally
Show answer
Answer: D. Pinned the base image digest and mirrored it internally
Public tags move as new patches ship, and old builds can disappear. Pinning base images by digest, keeping copies in an internal registry and storing the final image with the run make old environments recoverable.
60. Pods killed during a load test
Under load, Java pods restart with reason OOMKilled. The container memory limit is 2 GiB, and the JVM is started with -Xmx2g. What is the problem?
- A.The JVM needs a larger CPU limit to use memory
- B.Heap plus off-heap memory exceeds the 2 GiB container limit
- C.OOMKilled means the database ran out of disk
- D.The load test sent too many cookies
Show answer
Answer: B. Heap plus off-heap memory exceeds the 2 GiB container limit
The JVM uses memory beyond the heap: metaspace, thread stacks, code cache and direct buffers. With -Xmx equal to the container limit, total usage goes over it and the kernel kills the container. Set heap to a share of the limit (for example with MaxRAMPercentage) and leave room for the rest.