Skip to content

Scheduling and Placement

Every pod ends up on exactly one node, and that decision is permanent -- Kubernetes never moves a running pod. Scheduling is where you influence that one-shot decision.

This page covers the scheduler's decision process and the five placement controls you will actually use: nodeSelector, node affinity, taints and tolerations, pod affinity/anti-affinity, and topology spread constraints.

How the scheduler decides

For each unscheduled pod, the scheduler runs a two-phase cycle:

flowchart LR
    P[Unscheduled pod] --> F[Filtering\nwhich nodes are feasible?]
    F --> S[Scoring\nwhich feasible node is best?]
    S --> B[Bind\nwrite nodeName to the pod]
    F -->|zero feasible nodes| PEND[Pod stays Pending\nFailedScheduling event]
  1. Filtering eliminates nodes that cannot run the pod: insufficient unrequested CPU or memory, untolerated taints, failed affinity rules, volume topology conflicts, port clashes. Hard rules live here.
  2. Scoring ranks the survivors: spread pods across nodes, prefer nodes where the image is already pulled, honor preferred (soft) affinity rules, balance resource utilization. Preferences live here.
  3. Binding writes the winner into the pod's spec.nodeName. From that moment, placement is fixed -- the only way to "reschedule" a pod is to delete it and let its controller create a replacement.

Two consequences follow directly:

  • Scheduling uses requests, not usage. A node running at 20% actual CPU can still be infeasible if its requested capacity is exhausted. See Resource Requests and Limits.
  • Placement decisions do not self-correct. If you add affinity rules or new nodes, existing pods stay where they are until something recreates them. Tools like the descheduler exist precisely because the scheduler only acts at pod creation.

The direction of each control

The placement controls differ in who expresses the opinion:

Control Direction Hard or soft
nodeSelector pod chooses nodes hard
Node affinity pod chooses nodes both
Taints + tolerations node repels pods hard (with grace options)
Pod affinity / anti-affinity pod chooses based on other pods both
Topology spread constraints pod spreads across failure domains both

The mental model that keeps them straight: affinity is the pod asking to come in; a taint is the node keeping everyone out unless they carry a pass (a toleration). You need taints when the node must be protected from general workloads (GPU nodes, dedicated tenancy). You need affinity when the pod has requirements about where it lands.

A toleration alone does not attract a pod to a tainted node -- it only removes the barrier. Dedicated node pools therefore need both: a taint to keep everyone else off, plus a nodeSelector or node affinity so the intended pods actually go there.

nodeSelector and node affinity

nodeSelector is the simple form -- exact label match, hard requirement:

spec:
  nodeSelector:
    disktype: ssd

Node affinity is the expressive form. The field names are famously long, but they decode mechanically: when the rule applies (RequiredDuringScheduling = hard filter, PreferredDuringScheduling = soft score) and the honest admission that it is IgnoredDuringExecution -- already-running pods are never evicted when labels change.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["us-east-1a", "us-east-1b"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values: ["m6i.2xlarge"]

Semantics worth memorizing: multiple nodeSelectorTerms are OR; multiple matchExpressions within one term are AND. Operators include In, NotIn, Exists, DoesNotExist, Gt, Lt -- NotIn and DoesNotExist give you node anti-affinity.

Taints and tolerations

A taint is a key/value/effect triple on a node:

kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

The three effects escalate in strength:

  • NoSchedule: new pods without a matching toleration are filtered out. Existing pods are untouched.
  • PreferNoSchedule: soft version -- the scheduler avoids the node but will use it as a last resort.
  • NoExecute: also evicts already-running pods that do not tolerate the taint. A toleration can carry tolerationSeconds to delay that eviction.

The matching pod side:

spec:
  tolerations:
    - key: workload
      operator: Equal
      value: gpu
      effect: NoSchedule

NoExecute is not just an admin tool -- it is how Kubernetes handles node failure. When a node goes NotReady, the node controller adds node.kubernetes.io/not-ready:NoExecute and unreachable:NoExecute taints. Every pod carries an automatic toleration for these with tolerationSeconds: 300, which is exactly why pods on a dead node sit for five minutes before being replaced. Lowering that per-pod toleration is how you tune failover time for critical services.

This taint machinery also explains behavior you have already seen:

  • Control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule -- the reason your workloads never land there, and the toleration DaemonSets add when node agents must run everywhere.
  • The kubelet applies pressure taints (memory-pressure, disk-pressure) so the scheduler stops adding load to a struggling node.

Pod affinity and anti-affinity

These rules place pods relative to other pods, within a topology domain defined by topologyKey -- same node (kubernetes.io/hostname), same zone (topology.kubernetes.io/zone), or any node-label boundary.

The dominant production use is anti-affinity for high availability -- "replicas of this app must not share a node":

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: web
          topologyKey: kubernetes.io/hostname

Two cautions from production experience:

  • Required anti-affinity caps your replica count. With topologyKey: kubernetes.io/hostname and 5 nodes, the 6th replica is unschedulable forever. During rolling updates the surge pod needs a free node too. Prefer preferred anti-affinity or topology spread unless co-location is genuinely unacceptable.
  • Pod affinity ("run near the cache") couples availability. If the target pods are down or rescheduled, your pods may become unschedulable. Use preferred for latency optimizations.

Topology spread constraints

Topology spread is the modern, quantitative way to distribute replicas across failure domains -- it replaces most spreading uses of anti-affinity because it lets you bound the imbalance rather than forbid co-location outright:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: web

maxSkew: 1 means: across zones, the difference between the most and least loaded zone (counting matching pods) may not exceed one. whenUnsatisfiable: DoNotSchedule makes it a hard filter; ScheduleAnyway demotes it to a scoring preference.

The subtle failure mode: spread constraints count pods, including ones from old ReplicaSets during a rollout, and the constraint is only evaluated at scheduling time -- scale-downs can re-skew the distribution afterward. For most services, maxSkew: 1 + ScheduleAnyway gives good spreading without wedging rollouts; reserve DoNotSchedule for strict zonal-HA requirements.

Priority and preemption

PriorityClasses rank pods; when a high-priority pod cannot schedule, the scheduler may preempt (evict) lower-priority pods to make room:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-service
value: 1000000
globalDefault: false
description: "Revenue-critical services; may preempt batch work"

Preempted pods get their graceful termination period -- preemption is eviction, not kill-on-sight. The built-in classes system-cluster-critical and system-node-critical are why core components (CNI, DNS) survive resource pressure that evicts your apps.

Use a small, documented set of classes (for example: batch < default < critical). A proliferation of ad-hoc priorities turns scheduling into an arms race where every team claims to be critical.

Debugging placement

Placement problems always surface as Pending pods, and the scheduler tells you exactly why:

kubectl describe pod <pod> | grep -A5 Events
# 0/10 nodes are available: 4 Insufficient cpu,
# 3 node(s) had untolerated taint {workload: gpu},
# 3 node(s) didn't match pod affinity/anti-affinity rules.

Every node lands in exactly one bucket -- read the census, and the fix is usually obvious. See Troubleshooting for the full triage flow.

kubectl get nodes --show-labels
kubectl describe node <node> | grep -A5 Taints
kubectl get pods -o wide -l app=web        # verify actual spread

Certification notes

  • Taints/tolerations and node affinity are core CKA material. Practice kubectl taint nodes (and removing a taint with the trailing -: kubectl taint nodes n1 key:NoSchedule-).
  • Remember the direction: taints live on nodes, tolerations on pods; a toleration never attracts a pod.
  • Know the three taint effects and that NoExecute evicts running pods.
  • For exam speed: kubectl explain pod.spec.affinity.nodeAffinity --recursive reproduces the field structure you can never quite remember.

Key Takeaways

  • Scheduling is a one-time filter-then-score decision based on requests, not usage; it never revisits running pods.
  • Affinity is the pod's preference; taints are the node's defense. Dedicated pools need both.
  • Prefer soft rules (preferred, ScheduleAnyway) unless violation is genuinely unacceptable -- hard rules are how rollouts wedge.
  • Topology spread constraints are the right tool for zone/node HA distribution; anti-affinity is the right tool for strict "never together."
  • Every placement mystery is solved by reading the FailedScheduling event census.

Further Reading