Kubernetes Operator Scenario Interview Questions (2027)
40 scenario-based Kubernetes Operator interview questions: stuck finalizers, hot reconcile loops, RBAC, owner references, CRD versioning, leader election.
Scenario-based Kubernetes Operator interview questions describe a controller misbehaving in a real cluster, such as a resource stuck in Terminating, a reconcile loop running hot, a CRD upgrade that breaks old objects or two replicas fighting over the same resource, and ask how you would find the cause and fix it. Good answers name the mechanism (finalizers, owner references, status, leader election, conversion webhooks) and the command or code change. This page has 40 debugging and design scenarios grouped by experience level.
This page is the troubleshooting companion to the main question bank, 75+ Kubernetes Operators interview questions, and the six-week Operator interview prep plan. Background reading: how operators automate stateful applications and the role of operators for stateful apps.
Key takeaways
- Most operator incidents trace back to a few mechanisms: finalizers, owner references, status updates, watches and leader election.
- Reconcile must be idempotent. It should compute desired state from the current object every time, not from memory of previous calls.
- Returning an error requeues with backoff; returning
RequeueAfterschedules a check. Mixing these up causes hot loops or stalls. - Write status through the status subresource and use
observedGenerationso users can tell whether the controller has seen their latest change. - CRD version changes need a storage version plan and often a conversion webhook.
How the scenarios are organised
| Section | Scenarios | Focus |
|---|---|---|
| Freshers | 1 to 10 | Reading operator behaviour with kubectl |
| 2 to 5 years | 11 to 26 | Reconcile bugs, RBAC, watches, status, webhooks |
| Senior | 27 to 34 | CRD versioning, scale, upgrades, distribution |
| Production incidents | 35 to 40 | End-to-end incident walk-throughs |
Symptom to mechanism map
| Symptom | Likely mechanism | First check |
|---|---|---|
| Object stuck in Terminating | Finalizer never removed | kubectl get <kind> <name> -o jsonpath='{.metadata.finalizers}' |
| Controller CPU high, API server busy | Hot reconcile loop | Reconcile rate metrics, status writes triggering events |
| Child objects not cleaned up | Missing owner reference | .metadata.ownerReferences on children |
| Changes to children not corrected | Missing Owns() watch | Controller setup code |
| "forbidden" errors in logs | RBAC missing | kubectl auth can-i --as=system:serviceaccount:ns:sa |
| Two replicas acting at once | Leader election off or misconfigured | Lease object holder |
| Old objects fail after upgrade | CRD version or conversion issue | status.storedVersions on the CRD |
Freshers: reading operator behaviour
1. You created a custom resource and nothing happened.
Check that the operator pod is running and look at its logs. Confirm the CRD kind and API version match what the operator watches, and that the operator watches your namespace (some are installed for a single namespace). kubectl describe on the resource may show events or status conditions.
2. How do you see what the operator thinks of your resource?
Read .status, especially conditions such as Ready or Degraded, and compare status.observedGeneration with metadata.generation. If they differ, the controller has not processed your latest spec yet.
3. You deleted a custom resource and it stays in Terminating.
A finalizer is still present. The operator should run cleanup and remove it. Check operator logs for cleanup errors. Removing the finalizer by hand skips cleanup and can leave external resources behind, so do it only when you understand the consequences.
4. What is a CRD versus a custom resource?
The CustomResourceDefinition registers a new type with the API server, including its schema and versions. A custom resource is one object of that type.
5. You changed a Deployment the operator created, and your change disappeared.
That is expected. The operator owns the Deployment and reconciles it back to the state derived from the custom resource. Change the custom resource spec instead.
6. Where do you find which operators are installed?
List CRDs with kubectl get crd, look for operator Deployments, and on clusters with Operator Lifecycle Manager check Subscription and ClusterServiceVersion objects.
7. Your custom resource is rejected with a validation error.
The CRD's OpenAPI schema or CEL validation rules (x-kubernetes-validations) failed. Read the message, which names the field, and run kubectl explain <kind>.spec to see allowed fields.
8. The operator pod restarts often.
Check kubectl describe pod for OOMKilled or failed probes, and the logs of the previous container with kubectl logs --previous. Large caches of watched objects often explain memory pressure.
9. Why use an operator instead of a Helm chart?
Helm installs and upgrades a set of manifests. An operator keeps running and reacts to changes, so it can handle day-2 tasks such as backups, failover and version upgrades. Many teams use both. See the role of Helm.
10. How do you check operator events?
kubectl get events -n <ns> --field-selector involvedObject.name=<name> shows events the operator recorded against your resource, if it uses an event recorder.
2 to 5 years: reconcile, RBAC and watches
11. The controller reconciles the same object hundreds of times per minute.
Usually the controller updates the object (often status or an annotation) on every pass, which creates a new event and triggers reconcile again. Only write when something changed, compare before updating, and filter spec-only changes with GenerationChangedPredicate where appropriate. Check that you are not returning an error for a normal state.
12. Reconcile returns an error for "waiting for database to be ready".
Waiting is not an error. Return ctrl.Result{RequeueAfter: 30 * time.Second}, nil and set a condition explaining the wait. Errors trigger exponential backoff and noisy logs.
if !db.Ready() {
meta.SetStatusCondition(&obj.Status.Conditions, metav1.Condition{
Type: "Ready", Status: metav1.ConditionFalse,
Reason: "WaitingForDatabase", Message: "database not ready",
})
if err := r.Status().Update(ctx, obj); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}
13. Logs show "forbidden: User system:serviceaccount:... cannot list secrets".
The operator's ClusterRole or Role lacks that permission. With Kubebuilder or Operator SDK, add an RBAC marker and regenerate manifests, then redeploy. Verify with kubectl auth can-i list secrets --as=system:serviceaccount:ns:sa.
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch
14. Someone deletes a child ConfigMap and the operator does not recreate it.
The controller is not watching ConfigMaps it owns. Add Owns(&corev1.ConfigMap{}) to the controller builder and set an owner reference on the ConfigMap with controllerutil.SetControllerReference, so events on the child map back to the parent.
15. Children are left behind after the parent is deleted.
Owner references are missing, or the child is in a different namespace (namespaced owners cannot own objects in other namespaces). Set owner references for same-namespace children; for cross-namespace or external resources, use a finalizer and clean up explicitly.
16. A conflict error appears: "the object has been modified".
Optimistic concurrency: you updated with a stale resourceVersion. Re-fetch and retry, use retry.RetryOnConflict, or use patches or server-side apply so you only send the fields you manage.
17. Status updates overwrite spec changes or fail.
Enable the status subresource on the CRD and write status with r.Status().Update or Patch. Writes to the main resource ignore status, and status writes ignore spec, which prevents clobbering.
18. The operator cannot see objects in other namespaces.
The manager's cache may be restricted to certain namespaces, or RBAC is namespaced. Check the cache options in manager setup and whether you need a ClusterRole.
19. You need to look up all custom resources that reference a given Secret.
Add a field indexer on the reference field and use Watches with a mapping function so a Secret change enqueues the dependent custom resources. Listing everything on every event does not scale.
20. The validating webhook blocks all creates after a certificate rotation.
The API server cannot verify the webhook serving certificate. Check the CA bundle in the webhook configuration (cert-manager's CA injector usually manages it), the Secret mounted in the operator pod and webhook service endpoints. Consider failurePolicy carefully for non-critical webhooks.
21. The cleanup in your finalizer fails because the external API is down.
Return an error so reconcile retries with backoff, keep the finalizer in place, and surface a condition saying cleanup is pending. Do not remove the finalizer until cleanup succeeds, or you leak the external resource.
22. Two operator replicas both create the same external resource.
Leader election is off. Enable it in the manager so only the replica holding the Lease runs controllers. Also make creation idempotent by checking for the resource first or using a deterministic name.
23. You need to test reconcile logic without a real cluster.
Use envtest from controller-runtime, which runs a real API server and etcd locally without nodes, and write tests that create resources and assert on results. Add end-to-end tests on kind for full behaviour.
24. A user changes a field that should be immutable.
Add a CEL rule in the CRD schema such as self == oldSelf on that field, which the API server enforces, or use a validating webhook for rules CEL cannot express.
storageClass:
type: string
x-kubernetes-validations:
- rule: "self == oldSelf"
message: "storageClass is immutable"
25. Metrics show reconcile duration spiking.
Controller-runtime exposes metrics such as controller_runtime_reconcile_time_seconds and workqueue_depth. Find slow external calls, add timeouts, and consider raising MaxConcurrentReconciles if work is independent per object.
26. The operator logs too little to debug.
Use structured logging with the object key on every line, log decisions (created, updated, skipped) at info and details at debug, and record Kubernetes events for actions users care about.
Senior: versioning, scale and distribution
27. You need to rename a field from v1alpha1 to v1beta1.
Serve both versions, mark one as the storage version, and add a conversion webhook that maps fields both ways. Kubebuilder uses a hub-and-spoke model for conversions. Test round trips so no data is lost.
28. You want to stop serving an old CRD version.
Check status.storedVersions on the CRD. Migrate stored objects to the new storage version (read and write each object, or use a migration tool), remove the old version from storedVersions, then stop serving it. Removing a version that still has stored objects breaks reads.
29. The operator manages 20,000 custom resources and uses too much memory.
Limit what the cache holds: restrict namespaces, use label selectors on watches, strip unneeded fields with cache transforms, and avoid watching high-volume types you do not need. Measure after each change.
30. How do you upgrade an operator without disrupting workloads?
Make reconcile backward compatible with existing resources, roll out with leader election so only one version acts at a time, test upgrades from previous releases in CI, and on OLM use channels and upgrade graphs.
31. How do you distribute your operator to other teams?
Package it as manifests or a Helm chart, or as an OLM bundle for clusters that run OLM (including OpenShift). OperatorHub.io lists community operators for OLM.
32. How do you decide what capability level to target?
The Operator Framework capability model runs from basic install through upgrades, full lifecycle, deep insights and auto pilot. Target the level your users need; backup and upgrade automation often matter more than auto-tuning.
33. Should your operator run one controller per CRD?
Usually yes. Each controller reconciles one primary kind and watches its children. That keeps logic simple and lets you tune concurrency per kind.
34. How do you secure an operator?
Least-privilege RBAC (avoid cluster-admin), run as non-root with a restricted Pod Security profile, scope Secret access, sign images, and validate user input in the CRD schema so the operator never acts on malformed specs.
Production incident scenarios
35. A namespace deletion hangs for an hour.
Check the namespace status conditions; they list remaining resources and finalizers. Often a custom resource has a finalizer and its operator was uninstalled first. Reinstall the operator so it can clean up, or after confirming external resources are handled, remove the finalizer. Uninstall operators after their resources in future.
36. After an operator upgrade, every managed database restarts.
The new version probably changed the generated pod template (a label, annotation or default), so StatefulSets rolled. Roll back the operator, diff generated manifests between versions, and add tests that fail on unintended template changes.
37. The API server reports high load and the culprit is your operator.
Look for a hot loop (Scenario 11), direct API calls that bypass the cache, or list calls without selectors. Use the cached client for reads, fix the update pattern and set client-side rate limits.
38. The operator deleted resources it should not have.
Check the selection logic: label selectors that match too broadly, or garbage collection from an owner reference set on the wrong object. Restore from backup, then add safeguards such as requiring a specific label and dry-run testing of deletes.
39. A CRD update was applied and now old custom resources cannot be read.
A schema or version change broke conversion. Reapply the previous CRD to restore reads, then fix the conversion webhook or schema and test against copies of real objects.
40. During a node outage, the operator's leader pod is gone and nothing reconciles.
Another replica takes over after the Lease expires (lease duration and renew deadline settings control how long). If only one replica runs, nothing reconciles until it reschedules. Run at least two replicas with leader election and pod anti-affinity.
How to prepare
- Build a small operator with Kubebuilder or Operator SDK on kind, then break it on purpose: drop an RBAC rule, forget
Owns(), update status every loop. - Practise reading CRD status, finalizers and Lease objects with kubectl.
- Explain each fix in under two minutes, naming the mechanism.
- Build strong cluster fundamentals with our CKAD training or CKA training, in Kharadi, Pune or live online.
FAQ
Why is my custom resource stuck in Terminating?
A finalizer has not been removed, usually because the operator's cleanup is failing or the operator is no longer running.
What causes a hot reconcile loop?
Updating the object on every reconcile, which creates a new watch event, or returning errors for normal waiting states.
Should I return an error or RequeueAfter?
Return an error for real failures that need backoff. Return RequeueAfter with no error when you are waiting for something expected.
Do I need a conversion webhook?
When you serve more than one CRD version with different schemas, yes. If schemas are identical, the None strategy is enough.
How do I test an operator?
Unit test logic, use envtest for controller tests against a real API server, and run end-to-end tests on kind.
Kubebuilder or Operator SDK?
Operator SDK builds on Kubebuilder for Go operators and adds OLM integration, plus Helm and Ansible operator types. Either works for interviews.
Official references
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0