Operators and CRDs¶

CRDs extend the Kubernetes API with new types. Operators add the controller logic that makes those types mean something.

Together, they let teams model and automate complex domain workflows directly in Kubernetes - and they are the pattern behind almost every tool in the Ecosystem section of this site. cert-manager's Certificate, Flux's Kustomization, Prometheus Operator's ServiceMonitor, Velero's Backup, Argo CD's Application: all the same two pieces, a CRD and a controller reconciling it.

The two halves¶

A CustomResourceDefinition is a schema registration. Applying one tells the API server "there is now a kind called Database in group example.com, here is its shape, store objects of it in etcd." That is all it does. Once installed, custom objects get the entire API machinery for free - kubectl get, watches, RBAC, admission control, audit logging, GitOps, resource quotas, kubectl explain.

An Operator is a controller process that watches those objects and reconciles them into concrete Kubernetes resources and real-world actions.

The split matters when things break: a CRD with no running controller behaves like a very expensive ConfigMap. Objects apply cleanly, kubectl get shows them, and absolutely nothing happens. "The resource applied but nothing was created" is nearly always a controller that is down, crash-looping, or lacking RBAC - not a CRD problem.

flowchart LR
  A[Custom Resource] --> B[Operator Controller]
  B --> C[StatefulSet, Service, Secret, PVC]
  C --> D[Observed State]
  D --> B

A complete CRD¶

Most CRD examples stop at kind and spec. Here is one with the parts that actually matter in production - versioning, validation, a status subresource, and printer columns:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # Must be <plural>.<group>
  name: databases.example.com
spec:
  group: example.com
  scope: Namespaced              # or Cluster
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames: [db]
    categories: [all]            # so `kubectl get all` includes it
  versions:
    - name: v1alpha1
      served: true               # API server accepts requests at this version
      storage: false             # but does not persist at it
      deprecated: true
      deprecationWarning: "example.com/v1alpha1 Database is deprecated; use v1"
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engineVersion: {type: string}
                size: {type: string}
    - name: v1
      served: true
      storage: true              # exactly one version may be the storage version
      subresources:
        status: {}               # splits status into its own write endpoint
        scale:                   # enables `kubectl scale` and HPA targeting
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
          labelSelectorPath: .status.selector
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engineVersion
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=="Ready")].status
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          required: [spec]
          properties:
            spec:
              type: object
              required: [engineVersion, storage]
              properties:
                engineVersion:
                  type: string
                  pattern: '^\d+\.\d+$'
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 9
                  default: 3
                storage:
                  type: string
                  # quantities are strings; validate the shape
                  pattern: '^[0-9]+(Mi|Gi|Ti)$'
                backupSchedule:
                  type: string
                  x-kubernetes-validations:
                    - rule: "self == '' || self.matches('^([^ ]+ ){4}[^ ]+$')"
                      message: "backupSchedule must be a five-field cron expression"
              x-kubernetes-validations:
                # CEL rules can see the whole object and the previous value
                - rule: "self.replicas % 2 == 1"
                  message: "replicas must be odd so the cluster can hold quorum"
                - rule: "self.storage == oldSelf.storage"
                  message: "storage is immutable once set"
            status:
              type: object
              properties:
                replicas: {type: integer}
                selector: {type: string}
                observedGeneration: {type: integer}
                conditions:
                  type: array
                  items:
                    type: object
                    required: [type, status]
                    properties:
                      type: {type: string}
                      status: {type: string}
                      reason: {type: string}
                      message: {type: string}
                      lastTransitionTime: {type: string, format: date-time}

Four things in there earn their keep:

openAPIV3Schema is your admission control for free. Every field you leave out of the schema is silently dropped on write (structural schemas prune unknown fields), and every constraint you express - required, minimum, pattern, enum - is enforced by the API server before your controller ever sees the object. x-kubernetes-validations adds CEL rules for the cross-field and immutability checks a plain schema cannot express. This is validation you do not have to write, test, or run a webhook for.

The status subresource splits the write path. With it enabled, a write to /databases/foo ignores changes to status, and a write to /databases/foo/status ignores changes to spec. That separation is what stops a controller writing status from clobbering a user's concurrent spec edit, and it is what makes metadata.generation useful: generation increments only on spec changes, so a controller comparing status.observedGeneration to metadata.generation knows precisely whether it has caught up.

additionalPrinterColumns is the difference between an operator people can debug and one they can't. kubectl get databases showing engine, replicas, and readiness is the first thing anyone runs during an incident.

scope is permanent in practice. Switching a CRD between Namespaced and Cluster after objects exist is not a supported migration - you export, delete the CRD (which deletes every object of that kind), and reapply.

Versioning and conversion webhooks¶

The CRD above serves two versions. The rule is that exactly one version is the storage version, and everything written at any served version is converted to the storage version before it hits etcd, then converted back on read.

With no conversion strategy specified, the default is None: the API server hands back the stored object with only its apiVersion field rewritten. That works only if the versions are structurally identical - a pure rename of the version string. The moment a field actually moves or changes shape, you need a webhook:

spec:
  conversion:
    strategy: Webhook
    webhook:
      conversionReviewVersions: ["v1"]
      clientConfig:
        service:
          name: database-operator-webhook
          namespace: database-system
          path: /convert
          port: 443
        caBundle: <base64 PEM>

Here is what a schema change to an existing stored object actually breaks, in the order people hit it:

1. Old objects stay stored at the old version. Changing storage: true to a new version does not rewrite anything already in etcd. Existing objects sit at the old storage version until someone writes them again. Until then, every read goes through conversion.
2. A conversion webhook that is down takes reads with it. Not just writes - if the stored version differs from the requested version, a plain kubectl get fails. An operator whose webhook Service has no endpoints can make its own CRs unreadable, including to the controller trying to fix them.
3. Removing a version too early is unrecoverable. You cannot drop the old version from spec.versions while objects are still stored at it, and status.storedVersions on the CRD is the record of which versions etcd still holds:

kubectl get crd databases.example.com -o jsonpath='{.status.storedVersions}'

The migration is: add the new version as served-but-not-storage, ship the conversion webhook, flip storage: true to the new version, rewrite every existing object (a no-op kubectl get -o yaml | kubectl apply -f - per object, or a storage-version migrator), confirm storedVersions lists only the new version, then remove the old one. 4. Field removal is data loss, and pruning makes it silent. If v2 drops a field v1 had, converting v1 to v2 and back does not restore it. Round-tripping through the storage version must be lossless, which in practice means new versions carry the old fields (possibly as annotations) until the old version is gone.

The reason to care even if you never write an operator: this is exactly why an operator upgrade can wedge a cluster. The new controller ships a new CRD version, the webhook Deployment is not ready yet, and every CR of that kind becomes unreadable in the gap.

The reconciliation contract¶

Whether you build operators or just operate them, the contract explains their behavior:

• Reconcile the whole state, not the event. A correct reconciler ignores what changed and asks "given the current desired state, what should exist?" This makes it naturally idempotent - running it twice does nothing the second time - and means missed events don't matter. Controllers use a work queue keyed by object name, and the queue deduplicates, so ten rapid changes may produce one reconcile. A reconciler that assumes it sees every intermediate state is already wrong.
• Expect to run again. Reconcilers re-run on every change, on periodic resync, and after controller restarts. Any action that isn't safe to repeat (sending an email, charging a card) doesn't belong in a reconcile loop without external deduplication.
• Status is the contract. Well-built operators report progress and errors in status and conditions. kubectl describe <cr> should tell you why something is stuck before you resort to operator logs.
• observedGeneration closes the loop. Without it, you cannot distinguish "the controller looked at this and it's healthy" from "the controller hasn't looked at this yet."

Requeue, backoff, and the hot loop¶

A reconcile returns one of three things, and picking wrong is the most common performance bug in operators:

Returning an error is not a failure mode to avoid - it is the mechanism that gives you backoff. Swallowing errors and returning "success, requeue in 5 seconds" instead converts a transient outage into fixed-rate polling that never backs off.

The pathological case is the status hot loop: the reconciler writes status on every pass, that write is a change to the object, the change wakes its own watch, and the reconciler runs again. If the written status differs each time - a timestamp, a re-ordered list, a recomputed float - it never converges, and one CR pins a CPU and hammers the API server. The fix is to compare before writing and skip the update when nothing changed semantically.

Symptoms from the outside, and what to check:

# Reconcile rate per controller -- a flat high plateau on an idle cluster is a hot loop
controller_runtime_reconcile_total
controller_runtime_reconcile_time_seconds_bucket
workqueue_depth
workqueue_adds_total

# Is the operator being rate-limited by the API server?
kubectl logs deploy/<operator> -n <ns> | grep -i "client-side throttling\|Throttling request"

Persistent client-side throttling messages usually mean the controller is issuing live GETs instead of reading from its informer cache.

ownerReferences and garbage collection¶

An operator creates a StatefulSet, a Service, a Secret, and some PVCs. When the CR is deleted, those must go too - and the mechanism is not operator code. It is ownerReferences:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-db
  namespace: prod
  ownerReferences:
    - apiVersion: example.com/v1
      kind: Database
      name: my-db
      uid: 8f3d2b1a-...        # the UID, not just the name
      controller: true
      blockOwnerDeletion: true

The garbage collector watches for owners that no longer exist and deletes their dependents. Three details cause real incidents:

• The uid is what's compared, not the name. Delete a CR and recreate it with the same name and the new object has a new UID - children still pointing at the old UID are orphans and get collected, sometimes minutes later, after the new CR has already adopted them by name.
• Owner references cannot cross namespaces, and a namespaced object cannot own a cluster-scoped one. A CR that provisions a ClusterRole cannot own it; that cleanup must be a finalizer instead.
• Missing ownerReferences leak silently. Nothing errors. The operator works, the CR deletes fine, and the StatefulSet, PVCs, and LoadBalancer Service it created stay forever - still costing money. Audit with:

kubectl get statefulset,svc,pvc -n prod \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name'

Deletion is cascading and background by default: the CR disappears immediately and dependents are cleaned up asynchronously. --cascade=foreground keeps the CR present until dependents are gone, which is what you want when deletion order matters.

Finalizers, and the deletion that never finishes¶

A finalizer is a string on metadata.finalizers that makes deletion a two-phase operation. kubectl delete on an object with finalizers does not delete it - it sets metadata.deletionTimestamp. The object stays in the API, now marked Terminating, until every finalizer has been removed by whoever owns it.

That is exactly what an operator needs to deprovision something outside the cluster - an RDS instance, a DNS record, a cloud load balancer - before the CR disappears and it loses the information needed to do so.

apiVersion: example.com/v1
kind: Database
metadata:
  name: my-db
  namespace: prod
  finalizers:
    - databases.example.com/deprovision-cloud-resources
  deletionTimestamp: "2026-09-08T14:22:10Z"   # set by the API server on delete

The failure mode is predictable and common: the operator is down, so nothing removes the finalizer, so the object hangs in Terminating forever. Worse, deleting the namespace hangs too - namespace deletion cannot complete while any object in it still has a finalizer, so the namespace itself sits in Terminating indefinitely.

Diagnose before you reach for the hammer:

kubectl get database my-db -n prod -o jsonpath='{.metadata.finalizers}'
kubectl get database my-db -n prod -o jsonpath='{.metadata.deletionTimestamp}'
kubectl get pods -n <operator-namespace>          # is the operator even running?
kubectl logs deploy/<operator> -n <operator-namespace> --tail=100

If the operator is merely down, start it - it will finish the cleanup and remove the finalizer itself. The escape hatch is for when the operator is gone for good:

kubectl patch database my-db -n prod \
  --type=merge -p '{"metadata":{"finalizers":null}}'

This is data loss, deliberately. Stripping the finalizer tells the API server the cleanup is done when it is not: the cloud database, the DNS record, the volume the operator was going to snapshot - all of it is now orphaned outside the cluster, still billing, with nothing left in the cluster referencing it. Before you run it, write down what the operator was supposed to clean up, because after you run it nothing in Kubernetes remembers. Use it when the operator has been uninstalled and its CRD is on its way out, not as a routine unstick.

Operator RBAC creep¶

This is the governance problem that gets least attention and matters most.

An operator is a controller with a ServiceAccount, and to create StatefulSets, Services, Secrets, and PVCs it needs permission to do so cluster-wide. Many ship a ClusterRole close to cluster-admin, sometimes literally:

# The pattern to reject in review
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]

Three distinct risks compound here:

1. The operator is a privilege-escalation path. Anyone who can create a CR can make the operator act with the operator's permissions. If the operator can create Secrets in any namespace and read them back, then the ability to create a Database in your own namespace is, transitively, the ability to reach far beyond it. This is the same class of problem as Argo CD's AppProject restrictions or Flux's serviceAccountName impersonation - a controller with broad rights is a delegation of those rights to whoever can queue work for it.
2. A compromised operator pod is a compromised cluster. Its ServiceAccount token is mounted in the pod. An RCE in an operator with secrets: ["*"] reads every Secret in the cluster.
3. Under-permissioned operators fail silently and confusingly. The mirror image, and far more common day to day: an operator that can create Deployments but not PodDisruptionBudgets reconciles most of the way and then fails on one object. The CR shows a partial status, the app half-works, and the actual error is buried in operator logs as a forbidden line nobody thought to look for.

What to do:

# What can this operator actually do?
kubectl get clusterrole <operator-role> -o yaml
kubectl auth can-i --list \
  --as=system:serviceaccount:<ns>:<operator-sa>

# The check that matters most
kubectl auth can-i create clusterrolebindings \
  --as=system:serviceaccount:<ns>:<operator-sa>
kubectl auth can-i get secrets -A \
  --as=system:serviceaccount:<ns>:<operator-sa>

# Find the silent-failure case
kubectl logs deploy/<operator> -n <ns> | grep -i "forbidden\|cannot \(create\|list\|watch\)"

Practical rules for adopting a third-party operator:

• Read the ClusterRole before you install the chart. It is in the Helm chart or bundle, and reviewing it takes two minutes.
• Prefer namespace-scoped operators where the project supports it (watching a fixed set of namespaces via a Role per namespace rather than a ClusterRole). Many operators support this and default to cluster-wide anyway.
• escalate and bind are the verbs to grep for. RBAC normally prevents a subject from granting permissions it does not itself hold; those two verbs turn that off, and an operator holding either can grant itself anything.
• Treat operator upgrades as control-plane changes, because a new chart version can quietly widen the ClusterRole. Diff RBAC across upgrades.

Common failure modes¶

CRD vs aggregated API server¶

A CRD is not the only way to add a type. The alternative is an aggregated API server: your own API server process, registered via APIService, that the kube-apiserver proxies to. Metrics Server is the one nearly every cluster runs.

Choose a CRD unless you need something a CRD structurally cannot do: serving data that does not live in etcd (live metrics), custom subresources with behavior, or storage requirements etcd cannot meet. In practice that is a short list, which is why virtually every project in the ecosystem uses CRDs.

Building operators¶

If you are writing one rather than operating one:

• controller-runtime is the library underneath nearly everything - managers, informer-backed caches, work queues, leader election, and the Reconcile(ctx, req) (Result, error) signature the contract above describes. Kubebuilder scaffolds a project around it and generates CRD YAML from Go structs with // +kubebuilder:validation: markers, so the schema in this page's example is something you annotate rather than hand-write.
• Operator SDK wraps the same libraries and adds Helm-based and Ansible-based operators for cases where the reconcile logic is really just "render this chart," plus the OLM bundle packaging Red Hat's ecosystem uses.
• Leader election is not optional. Run more than one replica for availability, but let exactly one reconcile at a time - concurrent reconcilers on the same object race, and Kubernetes gives you no locking.
• Test with envtest, which runs a real kube-apiserver and etcd binary against your controller. Testing a reconciler against a fake client tests your mocks; testing against a real API server catches the schema pruning, defaulting, and status-subresource behavior that a fake client does not model.

When to choose Helm vs Operator¶

• Choose Helm for packaging and straightforward lifecycle - install, upgrade, roll back, done.
• Choose an Operator when the system needs continuous, domain-specific reconciliation: failover, version-aware upgrades, backup orchestration, quorum management.

The test: if the interesting logic happens at install time, it is a chart. If the interesting logic happens at 3am on a Tuesday when a primary fails, it is an operator.

Many platforms use both - Helm to install the operator, the operator to manage the application lifecycle.

Operational checks¶

kubectl get crd
kubectl get crd <name> -o jsonpath='{.spec.versions[*].name}{"\n"}{.status.storedVersions}'
kubectl get <custom-resource-plural> -A
kubectl describe <custom-resource-kind> <name> -n <namespace>
kubectl explain <custom-resource-kind>.spec --recursive
kubectl logs deploy/<operator-deployment> -n <operator-namespace>
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<operator-sa>

Governance considerations¶

• Review the CRD schema before adoption - a type with no validation and no status conditions will be undebuggable in production.
• Constrain operator RBAC to least privilege, and re-diff it on every upgrade.
• Monitor operator availability like a control-plane dependency: a down operator means stuck deletions and unreconciled state, not just a missing feature.
• Validate backup and restore behavior before production reliance, including what happens to CRs and their finalizers during a restore.

Summary¶

CRDs and Operators make Kubernetes extensible and automatable for advanced platforms. Use them when lifecycle logic is too complex for static manifests alone - and when adopting someone else's operator, read its CRD schema and its ClusterRole before you read its README.

Further Reading¶

• Kubernetes: Custom Resources
• Kubernetes: Extend the Kubernetes API with CustomResourceDefinitions
• Kubernetes: CRD versioning and conversion webhooks
• Kubernetes: validation rules (CEL)
• Kubernetes: owners and dependents (garbage collection)
• Kubernetes: using finalizers to control deletion
• Kubernetes: operator pattern
• Kubernetes: aggregation layer
• controller-runtime and Kubebuilder book
• Operator SDK
• OperatorHub

Related Concepts¶

• StatefulSets - what most database operators actually create
• RBAC - the model behind the operator-permissions section above
• cert-manager - a canonical CRD-plus-controller design
• Flux CD - multiple controllers, each owning one CRD family
• Argo CD - one controller, one Application CRD
• Maintenance - operator upgrades as control-plane changes
• Security Primer