DNS and Service Discovery¶

Every Service name your applications use -- db, api.backend, kafka-0.kafka.events.svc.cluster.local -- is resolved by the cluster's own DNS. When it works, nobody thinks about it. When it misbehaves, it produces some of the most confusing symptoms in Kubernetes: intermittent 5-second latency spikes, names that resolve in one namespace but not another, and external lookups that hammer your resolver five times per query.

This page explains the machinery well enough that none of those surprise you.

The moving parts¶

flowchart LR
    POD[Pod\n/etc/resolv.conf] -->|nameserver 10.96.0.10| SVC[kube-dns Service\nClusterIP]
    SVC --> CD1[CoreDNS pod]
    SVC --> CD2[CoreDNS pod]
    CD1 -->|cluster names| K8S[kubernetes plugin\nwatches Services + EndpointSlices]
    CD1 -->|everything else| FWD[forward plugin\nupstream resolvers]

• CoreDNS runs as a Deployment (typically 2 replicas) in kube-system, exposed by the kube-dns Service at a fixed ClusterIP (commonly 10.96.0.10).
• The kubelet writes each pod's /etc/resolv.conf pointing at that ClusterIP -- that's the whole integration.
• CoreDNS answers cluster names itself from an in-memory view built by watching Services and EndpointSlices, and forwards everything else (example.com) to the upstream resolvers from the node.

Configuration lives in the coredns ConfigMap as a Corefile:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
}

The two lines that matter operationally: kubernetes cluster.local ... (which zone is answered from cluster state) and forward . /etc/resolv.conf (where everything else goes).

What gets a name¶

Two details worth knowing:

• DNS reflects readiness. A pod failing its readiness probe drops out of headless-service DNS answers, just as it drops out of EndpointSlices. "DNS returns nothing" often means "no pod is ready," not "DNS is broken."
• Regular pods do not get useful DNS names. Only StatefulSet pods (via their headless Service) have stable per-pod records. Anything that needs to address individual replicas by name belongs in a StatefulSet.

The search path and ndots:5 -- read this section twice¶

The kubelet writes a resolv.conf like this into every pod:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search line is why short names work at all:

• db → tried as db.production.svc.cluster.local first → same-namespace lookup works
• api.backend → eventually tried as api.backend.svc.cluster.local → cross-namespace lookup works

The ndots:5 line is the famous footgun. It means: if a name contains fewer than five dots, try every search suffix before trying the name as-is. Consider a pod looking up api.stripe.com (two dots, fewer than five):

api.stripe.com.production.svc.cluster.local   → NXDOMAIN
api.stripe.com.svc.cluster.local              → NXDOMAIN
api.stripe.com.cluster.local                  → NXDOMAIN
api.stripe.com                                → finally, the real answer

Every external lookup costs four queries instead of one -- multiplied by every connection in a chatty service, this is a real source of CoreDNS load and tail latency. Mitigations, in order of preference:

1. Use a trailing dot for external names in config: api.stripe.com. is fully qualified and skips the search list entirely.

2. Lower ndots per pod when a workload talks mostly to external services:

   spec:
     dnsConfig:
       options:
         - name: ndots
           value: "2"
   

3. NodeLocal DNSCache: a DaemonSet cache on every node that absorbs repeated queries and keeps them off CoreDNS entirely. Standard equipment on large or DNS-heavy clusters.

The related classic symptom -- intermittent exactly-5-second DNS delays -- is a resolver timeout retrying a lost UDP query (historically aggravated by a kernel conntrack race with parallel A/AAAA lookups). NodeLocal DNSCache is the standard fix; forcing TCP (use-vc) is the fallback.

dnsPolicy: who answers this pod's queries¶

The naming is genuinely unfortunate: Default is not the default. ClusterFirst is. The most common real-world bug from this table: a hostNetwork: true pod silently getting node DNS (because hostNetwork implies node resolv.conf under ClusterFirst... unless you set ClusterFirstWithHostNet) and failing to resolve Services.

Debugging DNS, in order¶

# 1. Is it DNS, or is it the service? Resolve first, connect second.
kubectl run dnstest --rm -it --image=busybox:1.36 --restart=Never -- nslookup api.backend

# 2. Does the FQDN work when the short name doesn't? → search-path problem.
kubectl exec -it <pod> -- nslookup api.backend.svc.cluster.local

# 3. What does the pod's resolver config actually say?
kubectl exec -it <pod> -- cat /etc/resolv.conf

# 4. Is CoreDNS itself healthy?
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# 5. Can the pod even reach the DNS ClusterIP? (NetworkPolicy is a classic culprit)
kubectl exec -it <pod> -- nc -zu 10.96.0.10 53

Step 5 deserves emphasis: the moment you apply egress NetworkPolicies, you must explicitly allow UDP+TCP 53 to kube-system, or every name lookup in the namespace dies. It is the single most common self-inflicted DNS outage.

If CoreDNS logs show i/o timeout on forwarded queries, the problem is upstream (node resolvers, VPC DNS), not Kubernetes.

Certification notes¶

• Know the FQDN forms cold: <svc>.<ns>.svc.cluster.local and <pod>.<svc>.<ns>.svc.cluster.local for StatefulSets. Exam tasks ask you to wire one service to another by DNS name.
• kubectl run tmp --rm -it --image=busybox --restart=Never -- nslookup <name> is the standard exam verification one-liner.
• CoreDNS troubleshooting (deployment in kube-system, its ConfigMap Corefile, the kube-dns Service) is CKA material.

Key Takeaways¶

• CoreDNS answers cluster names from watched API state and forwards the rest; the kubelet wires pods to it via resolv.conf.
• Short names work because of the search path; ndots:5 makes external lookups expensive -- trailing dots, lower ndots, or NodeLocal DNSCache fix it.
• DNS answers reflect readiness -- an empty answer usually means no ready endpoints.
• dnsPolicy: Default is not the default, and hostNetwork pods need ClusterFirstWithHostNet.
• After enabling egress policies, allow port 53 first, ask questions later.

Related Concepts¶

• Services -- what the resolved ClusterIP actually is
• Networking Overview -- the full request path
• Network Policies -- the DNS egress rule every namespace needs
• StatefulSets -- per-pod DNS identity