Learn how you can troubleshoot DNS issues within Kubernetes clusters along with some great pointers on the way. Read on to learn more.
We recently upgraded one of our clusters to the latest version of Kubernetes, something that we have already been running on several other, much larger Kubernetes clusters with no issues. However, since the upgrade, we started seeing some bizarre networking patterns with delays. The thing is, though, none of them seemed reproducible, they all felt like DNS issues, such as delays in starting a connection that was exactly 60 seconds.
As a result, the next step for us was to start looking into trying to reproduce this issue. In order to be able to replicate requests inside the environment easily, we started a shell inside a container inside the cluster, which made requests to a few different services. We noticed that some requests were not failing with no particular consistency. Instead, they were responding with SERVFAIL from CoreDNS. For us, we saw it as a good sign because we finally had a specific failure that we could begin looking at more deeply.
We enabled logging inside CoreDNS in the appropriate ConfigMap, and once we completed the process, we started seeing a stream of logs with all the DNS requests. Therefore, we were able to start resolving things again, allowing us to understand how CoreDNS logged SERVFAIL inside finally. Yet another good sign, this helped us narrow down the specific issue. We also noticed that it was the only particular pod that was responding with SERVFAIL.
for i in .kubectl -n kube-system get pods -l k8s-app=kube-dns -o name.; do echo $i $(kubectl -n kube-system logs $i | grep -c SERVFAIL); done; pod/coredns-b7bbc9747-fbb7f 143 pod/coredns-b7bbc9747-zjrjb 0
At this point, we were able to narrow down the issue to a specific pod. To confirm, we started sending requests to the direct IP address of the pod to see if we could get a SERVFAIL without the clusterIP. Therefore, we eliminated any possibility that this had anything to do with kube-proxy up to this point. As expected, at some point, the pod started responding with SERVFAIL for a few moments until it began to react again.
We believe it's always important to take a step back and review the current issue, listing all the discoveries in order to determine the next troubleshooting steps. So at this point:
- Intermittent DNS resolution failures with SERVFAIL replies
- One CoreDNS pod is responding with some SERVFAIL; the other pods are OK
