Kubernetes DNS Lookup Timeouts — The ndots:5 Pitfall and How to Fix It

TL;DR — Quick Fix

# Add to your pod spec to reduce unnecessary DNS queries
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen
        value: ""

This single change can reduce DNS query volume by 50-80% and fix intermittent timeout issues.

---

The Problem: Why DNS Is Slow in Kubernetes

Every pod in Kubernetes gets this /etc/resolv.conf:

cat /etc/resolv.conf

nameserver 10.96.0.10
search my-namespace.svc.cluster.local svc.cluster.local cluster.local us-east-1.compute.internal
options ndots:5

The ndots:5 setting means: If a hostname has fewer than 5 dots, Kubernetes appends each search domain and tries those first before attempting the original name.

api.stripe.com has 2 dots (less than 5), so Kubernetes generates 4-5 failed DNS queries before finally resolving the actual hostname. Each failed query takes 5-30ms. That's 20-150ms of wasted latency on every external DNS lookup.

The Impact at Scale

For a microservice making 100 external API calls per second:

• Default ndots:5: 100 calls x 4 extra queries = 400 wasted DNS queries/sec
• With CoreDNS under load, this causes timeouts and 5-second delays

Symptoms you'll see:

• Intermittent 5-second request timeouts (DNS timeout default)
• dial tcp: lookup api.example.com: i/o timeout errors
• High p99 latency with unexplained spikes
• CoreDNS pods showing high CPU/memory

Fix 1: Set ndots Per Pod (Fastest Fix)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"
          - name: single-request-reopen
            value: ""
      containers:
        - name: app
          image: my-app:latest

Why ndots: "2"?

• Internal services: my-service.namespace.svc.cluster.local has 4 dots — still resolves via search path
• External services: api.stripe.com has 2 dots — resolves directly without search suffix attempts
• Short service names: redis has 0 dots — still resolves via search path

Why single-request-reopen?

• Linux sends A (IPv4) and AAAA (IPv6) queries on the same socket by default
• Some conntrack implementations lose the second response, causing 5-second timeouts
• single-request-reopen opens a new socket for each query, preventing this race

Fix 2: Use FQDN in Application Code

Append a trailing dot to bypass search suffix entirely:

env:
  - name: DATABASE_HOST
    value: "postgres.database.svc.cluster.local."  # Trailing dot = absolute
  - name: REDIS_HOST
    value: "redis.cache.svc.cluster.local."
  - name: EXTERNAL_API
    value: "api.stripe.com."  # Trailing dot = no search suffix

The trailing dot tells the resolver "this is already a fully qualified domain name — don't append search suffixes."

Fix 3: Deploy NodeLocal DNSCache

NodeLocal DNSCache runs a DNS cache on every node, reducing CoreDNS load and eliminating network hops for cached queries:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
    spec:
      priorityClassName: system-node-critical
      hostNetwork: true
      dnsPolicy: Default
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
        - effect: NoSchedule
          operator: Exists
        - effect: NoExecute
          operator: Exists
      containers:
        - name: node-cache
          image: registry.k8s.io/dns/k8s-dns-node-cache:1.23.1
          resources:
            requests:
              cpu: 25m
              memory: 5Mi
            limits:
              memory: 30Mi
          args:
            - "-localip"
            - "169.254.20.10,10.96.0.10"
            - "-conf"
            - "/etc/Corefile"
            - "-upstreamip"
            - "10.96.0.10"
          ports:
            - containerPort: 53
              name: dns
              protocol: UDP
            - containerPort: 53
              name: dns-tcp
              protocol: TCP

Benefits:

• DNS queries resolved locally on the node (sub-millisecond)
• CoreDNS load reduced by 80-90%
• Eliminates conntrack race conditions (local queries don't use conntrack)
• Cache survives CoreDNS restarts

Fix 4: Tune CoreDNS Performance

Scale CoreDNS Horizontally

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dns-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: autoscaler
          image: registry.k8s.io/cpa/cluster-proportional-autoscaler:v1.8.9
          command:
            - /cluster-proportional-autoscaler
            - --namespace=kube-system
            - --configmap=dns-autoscaler
            - --target=deployment/coredns
            - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"min":2,"max":10}}

Optimize CoreDNS Corefile

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        cache 60 {
            success 9984 60
            denial 9984 10
        }
        forward . /etc/resolv.conf {
            max_concurrent 1000
            policy sequential
        }
        loop
        reload
        loadbalance
    }

Key CoreDNS Tuning Parameters

Fix 5: Prevent the conntrack Race Condition

The infamous "5-second DNS timeout" in Kubernetes is caused by a Linux kernel race condition:

Pod sends A query     --+
Pod sends AAAA query  --+--> Same UDP socket --> conntrack --> CoreDNS
                        |
                        +-- conntrack only tracks ONE reply
                            Second reply gets dropped
                            5-second timeout before retry

Solutions (pick one):

# Option A: single-request-reopen (per pod)
dnsConfig:
  options:
    - name: single-request-reopen
      value: ""

# Option B: Disable IPv6 lookups if not needed
initContainers:
  - name: disable-ipv6-dns
    image: busybox:1.36
    command: ['sysctl', '-w', 'net.ipv6.conf.all.disable_ipv6=1']
    securityContext:
      privileged: true

Diagnosing DNS Issues

Check DNS Resolution Time

# From inside a pod
kubectl exec -it <pod> -- sh

# Time a DNS lookup
time nslookup api.stripe.com
time nslookup kubernetes.default.svc.cluster.local

# Verbose DNS resolution showing all queries
dig +search +showsearch api.stripe.com

Check CoreDNS Health

# CoreDNS pod status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS metrics
kubectl port-forward -n kube-system svc/kube-dns 9153:9153
curl http://localhost:9153/metrics | grep coredns_dns_request

Monitor DNS Latency with Prometheus

# Average DNS response time (p99)
histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket[5m]))

# NXDOMAIN rate (indicates search suffix overhead)
rate(coredns_dns_responses_total{rcode="NXDOMAIN"}[5m])
/ rate(coredns_dns_responses_total[5m])

# Cache hit ratio
rate(coredns_cache_hits_total[5m])
/ (rate(coredns_cache_hits_total[5m]) + rate(coredns_cache_misses_total[5m]))

---

Frequently Asked Questions

What does ndots:5 mean in Kubernetes?

ndots:5 means if a hostname has fewer than 5 dots, the resolver will first try appending each search domain from /etc/resolv.conf before trying the original name. Since most external domains have 1-3 dots, Kubernetes generates 4-5 useless DNS queries before finding the answer. This adds 20-150ms latency per external lookup.

Is it safe to change ndots to 2?

Yes, for most workloads. Internal Kubernetes service names with the full FQDN (service.namespace.svc.cluster.local) have 4 dots and will resolve correctly with ndots:2. Short service names (just "redis" or "postgres") have 0 dots and will still use the search path. The only edge case is partial multi-segment service names without the full FQDN.

What causes the 5-second DNS timeout in Kubernetes?

A Linux kernel conntrack race condition. When a pod sends A (IPv4) and AAAA (IPv6) DNS queries simultaneously on the same UDP socket, conntrack may only track one response. The second response gets dropped, and the application waits 5 seconds for the default DNS timeout before retrying. Fix with single-request-reopen in dnsConfig or deploy NodeLocal DNSCache.

How does NodeLocal DNSCache help?

NodeLocal DNSCache runs a caching DNS proxy on every node (via DaemonSet). Pods query the local cache instead of the cluster CoreDNS service. This eliminates network hops, avoids conntrack issues (local traffic bypasses conntrack), reduces CoreDNS load by 80-90%, and provides sub-millisecond cached responses.

How do I know if DNS is causing my application latency?

Check these indicators: high p99 latency with sudden 5-second spikes, i/o timeout errors mentioning DNS lookups, high NXDOMAIN rate in CoreDNS metrics, and CoreDNS pods showing elevated CPU. Use dig +search +showsearch from inside a pod to see all DNS queries generated for a single lookup.

---

Related Resources

• Kubernetes Pod Troubleshooting Guide — Debug common pod failures
• Kubernetes Ingress NGINX Setup — Networking and routing
• Linux Performance Troubleshooting — System-level network debugging