Kubernetes Operator Engineer Interview: Coding Round (2027)

36 Kubernetes Operator engineer interview questions on controller-runtime Reconcile code, owner references, status, finalizers, RBAC markers and envtest.

Sep 27, 2025 - 12:43
Nov 5, 2026 - 16:40
 0  60
Kubernetes Operator Engineer Interview: Coding Round (2027)

Kubernetes Operator engineer interviews include a coding round on controller-runtime: writing a Reconcile function that handles a missing object, creates owned children with owner references, updates status without conflicts, uses finalizers for external cleanup and requeues correctly. Interviewers also ask how you test with envtest and how you keep an Operator safe under load. This page has 36 questions for engineers who write Operators in Go, grouped by level, with a code-review scenario section.

It assumes you know the basics. If CRDs and reconcile loops are new to you, start with the most asked Operator questions for freshers. For a full question bank see the 75+ Kubernetes Operators interview questions, and for architecture-level topics see the advanced Operator questions. Code here follows the Kubebuilder book and controller-runtime APIs as of November 2026 (Kubebuilder v4.16.0, controller-runtime v0.25.2, Operator SDK v1.42.3).

Key takeaways

  • Return client.IgnoreNotFound(err) when the object was deleted. Returning an error requeues with backoff; returning ctrl.Result{RequeueAfter: d} schedules a planned recheck.
  • Set owner references with controllerutil.SetControllerReference and declare Owns() so child changes trigger the parent.
  • Update status through the status subresource and record observedGeneration. Handle 409 conflicts by re-reading, not by forcing.
  • Finalizer pre-delete logic must be idempotent, and the finalizer must be removed only after cleanup succeeds.
  • Generate RBAC from +kubebuilder:rbac markers and keep it minimal, including the /status and /finalizers subresources.
  • Test reconcile logic against a real API server with envtest before running it on a cluster.

What each round usually checks

RoundWhat you are asked to doWhat reviewers look for
Live codingWrite or fix a Reconcile functionNotFound handling, idempotency, owner references
API designDesign a CRD spec and statusValidation markers, conditions, versioning plan
Code reviewSpot bugs in a given controllerStatus conflicts, hot loops, missing RBAC
OperationsExplain how it runs in productionLeader election, metrics, upgrades, rate limiting

Core controller-runtime questions

1. What is the signature of a controller-runtime Reconcile function?

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error). The request holds only the namespaced name, not the object, so the first job is always to fetch the current object from the cache.

2. Write the first lines of a Reconcile function.

func (r *CacheReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := logf.FromContext(ctx)
    var cache appv1.Cache
    if err := r.Get(ctx, req.NamespacedName, &cache); err != nil {
        // deleted after the event was queued: nothing to do
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    log.Info("reconciling", "generation", cache.Generation)
    // ... build desired children, compare, create or update
    return ctrl.Result{}, nil
}

Ignoring NotFound stops pointless retries for objects that are already gone. Owned children are cleaned up by garbage collection.

3. What is the difference between returning an error and returning RequeueAfter?

Returning a non-nil error logs it, increments error metrics and requeues the request with exponential backoff from the workqueue rate limiter. Returning ctrl.Result{RequeueAfter: 30 * time.Second} with a nil error schedules a recheck at that time. Use errors for failures and RequeueAfter for expected waiting, such as polling an external API.

4. Why does the reconciler read from a cache?

The manager's client reads from informer caches to avoid load on the API server. Writes go straight to the API server. The cache can be slightly stale, so logic must tolerate seeing an older version and rely on conflicts and requeues to converge.

5. How do you register a controller that reacts to child changes?

func (r *CacheReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appv1.Cache{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Complete(r)
}

For sets the primary type. Owns enqueues the owner when an owned Deployment or Service changes, which depends on the controller owner reference being set.

6. How do you set an owner reference on a child?

Call controllerutil.SetControllerReference(&cache, &deploy, r.Scheme) before creating the child. It marks the parent as the controlling owner, enabling garbage collection and the Owns watch. Owners must be in the same namespace as namespaced children, or cluster-scoped.

7. What does controllerutil.CreateOrUpdate do, and what is its trap?

It gets the object, runs your mutate function, and creates or updates it if anything changed. The trap is a mutate function that overwrites the whole spec, wiping defaulted fields and causing an update on every reconcile. Mutate only the fields you own.

8. When would you use server-side apply instead?

When several actors manage different fields of the same object. Server-side apply tracks field ownership through managed fields, so your controller can claim only its fields with a fixed field manager name and the API server merges the rest.

9. How do you watch a resource you do not own?

Use Watches with handler.EnqueueRequestsFromMapFunc to map events to the parent requests, for example a Secret referenced by several Cache objects. Add a field indexer so the map function can list affected parents without scanning everything.

10. What is a field indexer and why use it?

mgr.GetFieldIndexer().IndexField registers a function that extracts a value, such as the referenced Secret name, from each object. You can then list with client.MatchingFields against the cache, which is fast and avoids a full list in every reconcile.

11. What are predicates?

Filters on events before they reach the queue. predicate.GenerationChangedPredicate ignores updates that do not change metadata.generation, such as status-only updates, which helps prevent reconcile loops triggered by your own status writes.

12. Why should the reconciler not depend on event type?

Requests carry only a name, and events can be merged or missed during restarts. Reconcile must compute the whole desired state from current objects each time. This is the level-based design Kubernetes controllers follow.

Status, finalizers and API design

13. How do you update status correctly?

Use r.Status().Update(ctx, &cache) or a status patch, with the status subresource enabled through +kubebuilder:subresource:status. Set status.observedGeneration to the generation you processed so users know whether status reflects the latest spec.

14. How do you handle a 409 conflict on update?

A conflict means the resourceVersion you read is outdated. Return the error so the request is retried with a fresh read, or use retry.RetryOnConflict around a get-modify-update for short operations. Never drop the resourceVersion to force a write.

15. How should conditions be written?

Use []metav1.Condition in status and meta.SetStatusCondition to update them. It changes lastTransitionTime only when the status value changes. Use positive types such as Ready or Available, with a machine-readable reason.

16. Write the finalizer flow.

const fin = "cache.example.com/cleanup"

if cache.DeletionTimestamp.IsZero() {
    if !controllerutil.ContainsFinalizer(&cache, fin) {
        controllerutil.AddFinalizer(&cache, fin)
        return ctrl.Result{}, r.Update(ctx, &cache)
    }
} else {
    if controllerutil.ContainsFinalizer(&cache, fin) {
        if err := r.deleteExternal(ctx, &cache); err != nil {
            return ctrl.Result{}, err // retry, keep finalizer
        }
        controllerutil.RemoveFinalizer(&cache, fin)
        return ctrl.Result{}, r.Update(ctx, &cache)
    }
    return ctrl.Result{}, nil
}

The Kubebuilder book stresses that the pre-delete logic must be idempotent, because it can run more than once.

17. Which RBAC markers does a finalizer need?

update on the resources=caches/finalizers subresource, in addition to the main resource verbs and caches/status. Kubebuilder scaffolds these three markers, and make manifests turns them into a ClusterRole.

18. How do you validate CRD fields?

Use markers such as +kubebuilder:validation:Minimum=1, Enum, Pattern and +kubebuilder:default, plus CEL rules with +kubebuilder:validation:XValidation for cross-field checks. Validation in the schema rejects bad input before it reaches your code, and needs no webhook.

19. When do you still need a validating webhook?

When a check needs data outside the object, such as whether a referenced Secret exists or quotas across objects. Webhooks add an availability dependency: if the webhook is down and its failure policy is Fail, writes to that resource are blocked.

20. How do you add a new API version?

Run kubebuilder create api for the new version, mark one version as storage with +kubebuilder:storageversion, and implement conversion. Kubebuilder uses a hub-and-spoke model: one hub version implements Hub(), others implement ConvertTo and ConvertFrom, served by a conversion webhook.

21. What goes in spec versus status in your API design?

Spec holds user intent that the controller must honour: size, version, references. Status holds observations: ready replicas, endpoints, conditions, observedGeneration. Anything the controller computes belongs in status, never written back into spec.

22. What are print columns?

+kubebuilder:printcolumn markers add columns to kubectl get, for example Ready condition and age. They are a small touch that shows interviewers you think about the people who operate the API.

Testing and production questions

23. What is envtest?

A package in controller-runtime that starts a real kube-apiserver and etcd locally, without nodes or kubelets. You install your CRDs, run the manager or call Reconcile directly, and assert on objects. Kubebuilder scaffolds envtest suites with Ginkgo and Gomega.

24. What can envtest not test?

There are no built-in controllers or nodes, so Deployments never create pods and garbage collection does not run. Test that your controller created a Deployment with the right spec and owner reference, not that pods became ready. Use kind or a real cluster for end-to-end tests.

25. How do you test idempotency?

Call Reconcile twice with the same object and assert that the second call makes no changes: same resourceVersions on children, no duplicate objects, and no new events.

26. What does MaxConcurrentReconciles change?

It sets how many reconciles run in parallel for one controller. The workqueue guarantees the same key is never processed by two workers at once, so raising it is safe for correctness but adds load on the API server and external systems.

27. Why enable leader election?

With several replicas for availability, only the leader should run controllers to avoid conflicting writes. The manager uses a Lease. Kubebuilder's generated main.go exposes a flag to turn it on.

28. Which metrics does a controller-runtime manager expose?

Reconcile totals, errors and durations per controller, workqueue depth and latency, and client request metrics. Alert on rising error rates and queue depth. You can register custom metrics with the controller-runtime metrics registry.

29. How do you upgrade an Operator without breaking existing resources?

Keep CRD changes backward compatible, add new fields as optional with defaults, serve old versions until stored objects are migrated, and roll out the new controller image after the CRD. With OLM, ship a new bundle and test it with operator-sdk run bundle-upgrade.

30. How do you limit the Operator's permissions?

Scope RBAC to the verbs and resources the code uses, prefer namespace-scoped Roles when the Operator watches specific namespaces, restrict cache scope in the manager options, and run the pod with a restricted Pod Security Standard profile.

Code-review scenarios

31. The controller updates status every reconcile and CPU is at 100%.

Each status write triggers a watch event that triggers another reconcile. Compare status before writing and skip unchanged updates, and add GenerationChangedPredicate on the primary type if status changes should not requeue.

32. Deleted custom resources leave cloud load balancers behind.

The external resource has no Kubernetes owner, so garbage collection cannot remove it. Add a finalizer with idempotent cleanup and remove it only after the cloud API confirms deletion.

33. The Operator logs "forbidden" for configmaps after a new feature.

The code started managing ConfigMaps without an RBAC marker. Add // +kubebuilder:rbac:groups="",resources=configmaps,verbs=get;list;watch;create;update;patch, run make manifests and redeploy the role.

34. Child Deployments drift and the Operator never fixes them.

The controller does not watch Deployments or did not set a controller owner reference. Add Owns(&appsv1.Deployment{}) and call SetControllerReference on creation.

35. Reconcile returns an error every 5 seconds while waiting for a database to start.

Waiting is not a failure. Set a Progressing condition and return ctrl.Result{RequeueAfter: 15 * time.Second} with a nil error, so logs and error metrics stay meaningful.

36. Two replicas of the Operator fight over the same objects.

Leader election is disabled. Enable it so only one replica runs controllers, and check that both replicas use the same lease name and namespace.

How to prepare

Work through the Kubebuilder CronJob tutorial end to end, then add a finalizer, a CEL rule and a second API version to your own project. Practise explaining each line of a Reconcile function out loud. The Operator interview preparation plan sets out a weekly schedule, the scenario Operator questions test production judgement, and how Operators automate stateful apps covers the database patterns many Operators implement.

For the Kubernetes skills underneath, the CKA training is available in our Kharadi, Pune classroom and live online.

Frequently asked questions

What should a Reconcile function return when the object is gone?

ctrl.Result{} with client.IgnoreNotFound(err), so the deleted object is not retried.

Is it better to return an error or RequeueAfter?

Return an error for real failures so backoff applies. Use RequeueAfter with a nil error for planned waiting.

How do I test an Operator without a cluster?

Use envtest, which runs a local API server and etcd. It does not run built-in controllers, so pods are not created.

Do I need webhooks for validation?

Not for most checks. OpenAPI markers and CEL rules in the CRD handle field and cross-field validation. Use webhooks for checks that need outside data.

Why does my controller loop forever?

Usually because it writes status or the child spec every time. Write only when something changed, and filter events with predicates.

Which versions should I learn?

Current releases are Kubebuilder v4.16.0, controller-runtime v0.25.2 and Operator SDK v1.42.3. Check release notes for API changes before you start.

References

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Angry Angry 0
Sad Sad 0
Wow Wow 0
Mridul I am a passionate technology enthusiast with a strong focus on DevOps, Cloud Computing, and Cybersecurity. Through my blogs at DevOps Training Institute, I aim to simplify complex concepts and share practical insights for learners and professionals. My goal is to empower readers with knowledge, hands-on tips, and industry best practices to stay ahead in the ever-evolving world of DevOps.