Docker Container DNS Problem - The Fix

The DNS Problem

I recently ran into a problem with my containers not being able resolve names to ip addresses. I found this problem when I was installing Jenkins in a container. When I got to the step to install plugins jenkins said it was not connected to the internet. So I do a couple of checks.

First I got the Container ID from docker.

# docker ps
CONTAINER ID        IMAGE                                                                                 COMMAND                  CREATED             STATUS                  PORTS                 NAMES
91aa53f4642a        jenkins@sha256:c0cac51cbd3af8947e105ec15aa4bcdbf0bd267984d8e    7be5663b5551bbc5f4b   "/bin/tini -- /usr..."   5 hours ago         Up 5 hours  

 Notice the Container ID is 91aa53f4642a now you can attach to the container and run any system command you want.

#docker exec -it 91aa53f4642a /bin/bashjenkins@91aa53f4642a:/$ ping www.google.com

The ping command returned not found. Next I checked if I could actually get to an external IP address.

jenkins@91aa54f4642a:/$ ping 8.8.8.8

When I ran this I got access to the remote site. So I have internet connectivity, but no name resolution.

The Fix

Turns out this is a known problem with docker 1.11 and on. When the resolv.conf is created for for the container it does its best to handle inter-container name and service resolution, but it does not handle the external name resolution. In order to make this happen the host machine of the docker container must start dockerd with the --dns option set to an external DNS like 8.8.8.8.

First you have to find out how dockerd is getting started. if you are using Linux this is probably in the systemctl subsystem. For CentOS you can find this by using the systemctl command.

# systemctl show docker.service | grep Fragment
FragmentPath=/usr/lib/systemd/system/docker.service

Now look at the docker.service file and look for the ExecStart. /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target firewalld.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd 
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target

Now edit this file and change the ExecStart to include the --dns option.

ExecStart=/usr/bin/dockerd --dns=8.8.8.8 

This will tell all of the containers that get started on this host to use 8.8.8.8 as the secondary dns service after the inter-container dns service. Now that you have made the change then you need to restart the docker daemon.

# systemctl daemon-reload
# systemctl restart docker.service

That is all you need to do. Now you check things out by running ping in the container again.

#docker exec -it 91aa53f4642a /bin/bashjenkins@91aa53f4642a:/$ ping www.google.com

Hope this helps you out with this problem.

DWP