Advanced Kubernetes Operator Interview Questions (2027)
Advanced Kubernetes Operator interview questions: reconcile semantics, finalizers, informer caches, CRD versioning, conversion webhooks, CEL validation.
Advanced Kubernetes Operator interviews go inside the controller. Expect questions on reconcile semantics, informer caches, finalizers and owner references, status conditions, CRD versioning with conversion webhooks, CEL validation, leader election, RBAC markers and testing with envtest. Interviewers want to hear how an operator fails and recovers, not only how it is scaffolded. This page has 40 questions on operator internals.
It is the internals-focused member of our operator set. If you need broad coverage first, read the 75+ Kubernetes Operators interview questions. Production incidents are covered in Kubernetes Operator scenario interview questions, and a study schedule is in the 6-week operator prep plan. Facts below were checked against kubernetes.io, the Kubebuilder book, the Operator SDK site and the controller-runtime package docs in November 2026.
Key takeaways
- Reconcile must be level-based and idempotent: read desired and observed state, act, and return. Never assume you saw every event.
- In controller-runtime, returning a non-nil error requeues with exponential backoff;
reconcile.TerminalErrorstops retries.Result.Requeueis now deprecated in favour ofRequeueAfter. - A CRD has exactly one storage version. Removing an old version requires it to be gone from
status.storedVersionsfirst. - CEL validation rules (
x-kubernetes-validations) are generally available since Kubernetes 1.29 and replace many validating webhooks. - The Operator SDK capability model has five levels, from Basic Install to Auto Pilot. Know which level your operator reaches and why.
Operator vocabulary at a glance
| Term | What it is | Typical interview probe |
|---|---|---|
| Informer and cache | List-watch client that keeps a local copy of objects | Why reads can be stale; how to read fresh data |
| Work queue | Deduplicating, rate-limited queue of object keys | Why one reconcile per key at a time |
| Owner reference | Metadata linking a child to its parent | Garbage collection and watch triggers |
| Finalizer | String on an object that blocks deletion until removed | Cleaning external resources |
| Status subresource | Separate endpoint for .status | Why spec and status updates are split |
| Conversion webhook | Service that converts between CRD versions | Serving v1alpha1 and v1 together |
Foundation questions (asked of every operator candidate)
1. What is the operator pattern in one sentence?
An operator is a custom controller that watches a custom resource and continuously drives the cluster toward the state that resource describes, encoding the operational knowledge a human administrator would otherwise apply by hand.
2. What does level-based reconciliation mean?
The controller reacts to the current state, not to the event that woke it. Reconcile receives only a namespace and name, reads the object, compares desired and observed state and fixes the difference. If three updates arrive while one reconcile runs, the queue collapses them and the next run sees the latest state. Edge-triggered logic ("on create, do X") breaks when events are missed.
3. Why must Reconcile be idempotent?
It may run many times for the same state: on resync, after a restart, after errors or because a child object changed. Running it twice must not create duplicate resources or repeat side effects. Use deterministic child names, create-or-update helpers and status checks before acting.
4. What happens in controller-runtime when Reconcile returns an error?
The Result is ignored and the request is requeued with exponential backoff. If the error is wrapped with reconcile.TerminalError (added in v0.15.0), it is logged and counted in metrics but not retried. Use terminal errors for invalid user input that will not fix itself.
5. Requeue or RequeueAfter?
Use RequeueAfter when you are waiting for something outside the cluster, such as a cloud resource becoming ready. The controller-runtime docs mark Result.Requeue as deprecated because it causes confusion; RequeueAfter greater than zero already implies a requeue.
6. Why are spec and status updated separately?
With the status subresource enabled, the main endpoint ignores .status changes and the /status endpoint ignores everything else. Users own spec, the controller owns status. It also lets you grant RBAC on status separately and avoids the controller overwriting a user's spec edit.
7. What are owner references used for?
They tell the garbage collector to delete children when the parent goes, and they let a controller watch child objects and map events back to the owner. A controller reference (controller: true) marks the single managing owner. Cross-namespace owner references are not allowed; a namespaced child cannot be owned by an object in another namespace.
8. What are the five operator capability levels?
From the Operator SDK documentation: Level 1 Basic Install, Level 2 Seamless Upgrades, Level 3 Full Lifecycle, Level 4 Deep Insights, Level 5 Auto Pilot. Most in-house operators sit at levels 1 to 3. Saying that honestly, and naming what level 4 would need (metrics, alerts, dashboards), is a good answer.
Questions for engineers with 2 to 5 years of experience
9. How do informers reduce API server load?
Each informer does one list, then a long-running watch, and serves reads from a local cache shared by all controllers in the manager. Reconcilers read from that cache instead of calling the API server each time. The cost is memory proportional to the objects watched, and reads can lag the server briefly.
10. How do you limit cache memory in a large cluster?
Watch only what you need: restrict namespaces in the manager's cache options, use label or field selectors per object type, and avoid watching high-churn types such as all Pods or Secrets cluster-wide. For objects you only read occasionally, use an uncached client or metadata-only watches.
11. How does a finalizer work, step by step?
The controller adds a finalizer string when it first sees the object. When a user deletes it, the API server sets deletionTimestamp but keeps the object. The controller sees that, cleans up external resources, then removes its finalizer, and the API server deletes the object. If the cleanup can never succeed, the object stays in Terminating forever, so cleanup code needs timeouts, clear status conditions and an escape procedure.
12. Owner reference or finalizer for cleanup?
Owner references for objects inside the cluster, because the garbage collector handles them. Finalizers for things outside Kubernetes (cloud databases, DNS records, buckets) or for ordering that garbage collection cannot express.
13. How should status conditions be designed?
Use the standard metav1.Condition shape: type, status (True, False, Unknown), reason, message, lastTransitionTime and observedGeneration. Common types are Ready, Progressing and Degraded. Set observedGeneration so users and tools can tell whether status reflects the latest spec.
14. What is the difference between generation and resourceVersion?
metadata.generation increases when spec changes (with the status subresource enabled). resourceVersion changes on any write, including status and metadata. Filter reconciles with a generation-changed predicate when status-only updates should not trigger work.
15. What causes "the object has been modified" conflict errors and how do you handle them?
The update carried a stale resourceVersion because someone else wrote first. Return the error so the request is requeued and re-read, or use a retry-on-conflict helper around a fresh get and update. Server-side apply with a field manager reduces conflicts for objects the controller fully owns.
16. How do you generate RBAC for a Kubebuilder controller?
With markers above the reconciler, for example // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete. Use groups="" for the core API group. Running the manifests target regenerates the ClusterRole. Review it: generated RBAC is only as narrow as your markers.
17. Why does an operator need leader election?
So that only one replica reconciles at a time when you run several for availability. controller-runtime uses a Lease object; standby replicas take over when the leader stops renewing. Without it, two replicas fight over the same objects.
18. How do you test a controller without a full cluster?
Use envtest, which starts a real kube-apiserver and etcd locally, installs your CRDs and lets you run the reconciler against them. It catches schema, RBAC-free logic and status issues. Note that envtest has no controllers such as the garbage collector or scheduler, so owner-reference deletion and Pod scheduling need an end-to-end test on kind or a real cluster.
19. What belongs in an operator's metrics?
controller-runtime already exposes reconcile counts, errors, durations and work queue depth. Add domain metrics such as managed instances by phase, backup age or failover count. Alert on sustained reconcile errors and queue depth growth, not single failures.
Senior and design questions
20. How do you evolve a CRD from v1alpha1 to v1?
Add v1 to spec.versions with its schema, keep v1alpha1 served, and if schemas differ set spec.conversion.strategy: Webhook with a conversion webhook. Mark v1 as the storage version (exactly one version can be). Migrate stored objects, confirm v1alpha1 is gone from status.storedVersions, set v1alpha1 served: false, wait, then remove it and its conversion code.
21. How do you migrate objects stored in the old version?
Rewrite every object so the API server persists it in the new storage version, for example kubectl get <plural> -A -o json | kubectl replace -f - as shown in the Kubernetes docs, or use the Storage Version Migration API where available. Then patch status.storedVersions only after confirming no objects remain in the old version.
22. What does a conversion webhook receive and return?
A ConversionReview in the apiextensions.k8s.io/v1 API, listing objects and a desired API version. It returns the converted objects. It must be lossless round-trip: converting v1 to v1alpha1 and back must not drop fields, which often means storing new fields in annotations on the old version.
23. When would you use CEL validation instead of a validating webhook?
For rules that only need the object itself: field relationships, formats, immutability using oldSelf transition rules. CEL rules run inside the API server, so there is no webhook to deploy, scale or secure, and no outage if your operator is down. Keep a webhook only for checks that need external data or other objects.
24. Why are webhooks a reliability risk?
The API server calls them synchronously. If the webhook is down and failurePolicy is Fail, creates and updates for matched objects fail, sometimes including the operator's own deployment. Scope webhooks narrowly with selectors, run several replicas with a PodDisruptionBudget, set short timeouts, and exclude system namespaces.
25. How do you design a reconcile loop for a stateful system like a database?
Break the work into small steps that each check state before acting: ensure config, ensure StatefulSet, ensure Services, check members are healthy, then perform at most one disruptive action (failover, version bump) per reconcile. Record progress in status so a restart resumes rather than repeats. Our explainer on how operators automate stateful apps covers the concepts.
26. How do you handle long-running external operations?
Never block Reconcile. Start the operation, store its ID in status, return with RequeueAfter, and on the next reconcile poll the result. Blocking ties up a worker and hides progress from users.
27. How do you scale an operator that manages thousands of objects?
Raise MaxConcurrentReconciles so different keys reconcile in parallel, narrow the cache, filter events with predicates, and make each reconcile cheap. If one operator still cannot keep up, shard by namespace or label across instances, each with its own leader election lock.
28. Cluster-scoped or namespace-scoped operator?
Namespace-scoped installs need fewer permissions and suit multi-tenant clusters where teams run their own copies. Cluster-scoped installs are simpler to operate but need cluster-wide RBAC. The CRD itself is always cluster-wide, so two copies at different versions can conflict over it; decide who owns CRD upgrades.
29. How should an operator be packaged and upgraded?
Ship CRDs, RBAC and the manager as versioned manifests or a Helm chart, or as an Operator Lifecycle Manager bundle. OLM manages installation and upgrades from catalogs; a newer OLM v1 is developed under the operator-controller project. Whatever the method, CRD upgrades must stay backward compatible because existing objects remain.
30. How do you secure an operator?
Least-privilege RBAC from tight markers, no wildcard verbs, run as non-root with a read-only root filesystem, protect the metrics endpoint, and treat any Secret the operator reads as sensitive. If the operator creates workloads, enforce Pod Security on what it creates, since users can otherwise escalate through the custom resource.
31. Build an operator or use Helm?
Helm templates and installs; it does not watch and react after install. If day-2 work (backup, failover, version upgrades with data migration) needs automation, an operator earns its maintenance cost. If not, a chart plus GitOps is simpler. Saying "we should not build one" with reasons is a valid senior answer.
Scenario questions
32. Custom resources are stuck in Terminating. What do you check?
Look at metadata.finalizers and the operator logs. Usually the operator is down, lacks RBAC to remove the finalizer, or cleanup of an external resource fails. Fix the cause and let the operator finish. Removing the finalizer by hand is the last resort and may leak external resources, so record what was skipped.
33. The operator hot-loops, reconciling the same object hundreds of times a minute. Why?
Typically it writes on every reconcile: updating status with a new timestamp or rewriting a child with fields the API server defaults, which triggers a new event. Compare before writing, set only fields you own, and add a generation-changed predicate where status changes should not trigger reconciles.
34. After upgrading the operator, old custom resources fail validation. What happened?
The new schema probably added a required field or tightened a rule with no default. Existing objects are checked again when updated. Fix by adding defaults, making the field optional, or using a CEL transition rule that only applies to new values. Treat CRD schema changes like public API changes.
35. Two operator replicas are both reconciling. How?
Leader election is disabled, or the replicas use different lease names or namespaces. Check the manager flags and the Lease object. Also check for a second installation in another namespace watching the same resources.
36. The operator's memory grows until it is OOMKilled. Where do you look?
Cache contents first: an informer on Secrets or Pods cluster-wide in a big cluster is a common cause. Use pprof or memory metrics, restrict the cache, switch to metadata-only watches for large objects, and remove unused watches.
37. Deleting the custom resource did not delete its Deployment.
The Deployment lacks an owner reference to the custom resource, or it is in another namespace where owner references cannot apply. Set the controller reference when creating children, or clean up with a finalizer if cross-namespace children are required.
38. A conversion webhook outage blocks all reads of the custom resource. How do you prevent a repeat?
Run the webhook with several replicas, a PodDisruptionBudget and readiness checks, and monitor its latency and error rate. Keep schemas convertible without a webhook where possible (strategy None when the schemas match), and plan version removal so the webhook can eventually be retired.
39. Reconciles are slow because each one calls a cloud API several times. What do you change?
Cache external state in status with a timestamp and only refresh it on a schedule using RequeueAfter, batch calls where the API allows, and respect provider rate limits with backoff. Separate fast in-cluster work from slow external checks.
40. A team asks you to review their operator before production. What is on your checklist?
Idempotent reconcile, status conditions with observedGeneration, finalizers with timeouts, owner references on children, narrow RBAC, leader election, envtest and end-to-end tests, CRD versioning plan, metrics and alerts, resource limits and a documented manual recovery path.
Related reading: 12 Kubernetes CRDs that make life easier and where admission controllers enforce policy. Operators are built on core Kubernetes API skills; our CKAD training covers them in the Kharadi, Pune classroom and live online.
Frequently asked questions
Do I need Go to pass an advanced operator interview?
Most operator roles use Go with controller-runtime through Kubebuilder or the Operator SDK. You should be able to read and write a reconciler in Go, although the design questions apply in any language.
What is the difference between Kubebuilder and the Operator SDK?
Kubebuilder scaffolds Go operators on controller-runtime. The Operator SDK builds on Kubebuilder for Go projects and adds Helm and Ansible operator types plus OLM integration.
Is Result.Requeue still valid in controller-runtime?
It still exists but is deprecated. The package docs recommend RequeueAfter when waiting for an external event.
When did CEL validation for CRDs become stable?
Validation rules using x-kubernetes-validations reached general availability in Kubernetes 1.29.
How many storage versions can a CRD have?
Exactly one version must be marked as the storage version at any time, although several versions can be served.
How is this page different from the 75+ operator questions page?
The 75+ page covers the whole topic. This page goes deep on controller internals, CRD versioning and failure handling for senior roles.
Official references
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0