Docker Container DNS Problem - The Fix

The DNS Problem

I recently ran into a problem with my containers not being able resolve names to ip addresses. I found this problem when I was installing Jenkins in a container. When I got to the step to install plugins jenkins said it was not connected to the internet. So I do a couple of checks.

First I got the Container ID from docker.

# docker ps
CONTAINER ID        IMAGE                                                                                 COMMAND                  CREATED             STATUS                  PORTS                 NAMES
91aa53f4642a        jenkins@sha256:c0cac51cbd3af8947e105ec15aa4bcdbf0bd267984d8e    7be5663b5551bbc5f4b   "/bin/tini -- /usr..."   5 hours ago         Up 5 hours  

 Notice the Container ID is 91aa53f4642a now you can attach to the container and run any system command you want.

#docker exec -it 91aa53f4642a /bin/bashjenkins@91aa53f4642a:/$ ping www.google.com

The ping command returned not found. Next I checked if I could actually get to an external IP address.

jenkins@91aa54f4642a:/$ ping 8.8.8.8

When I ran this I got access to the remote site. So I have internet connectivity, but no name resolution.

The Fix

Turns out this is a known problem with docker 1.11 and on. When the resolv.conf is created for for the container it does its best to handle inter-container name and service resolution, but it does not handle the external name resolution. In order to make this happen the host machine of the docker container must start dockerd with the --dns option set to an external DNS like 8.8.8.8.

First you have to find out how dockerd is getting started. if you are using Linux this is probably in the systemctl subsystem. For CentOS you can find this by using the systemctl command.

# systemctl show docker.service | grep Fragment
FragmentPath=/usr/lib/systemd/system/docker.service

Now look at the docker.service file and look for the ExecStart. /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target firewalld.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd 
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target

Now edit this file and change the ExecStart to include the --dns option.

ExecStart=/usr/bin/dockerd --dns=8.8.8.8 

This will tell all of the containers that get started on this host to use 8.8.8.8 as the secondary dns service after the inter-container dns service. Now that you have made the change then you need to restart the docker daemon.

# systemctl daemon-reload
# systemctl restart docker.service

That is all you need to do. Now you check things out by running ping in the container again.

#docker exec -it 91aa53f4642a /bin/bashjenkins@91aa53f4642a:/$ ping www.google.com

Hope this helps you out with this problem.

DWP

Cloud Aware Application Development in Multiple Environments.

With the shift from traditional Client Server Application Software to Cloud Aware Application many Software Engineers have found themselves dusting off old System Administration Books from college. With multiple services running on multiple machine or containers software engineers have to be able to manage their applications across more and more complex environments. As I have been talking to some of my customers I have found common pain points in managing these complex applications: 

• Consistency between environments
• Single point of failure services
• Differing environment requirements  (Not all environments are created equal)
• Managing multiple environments across multiple clouds

All of these factors and many more can lead to time wasted, applications being released into production before their time, or worst of all unhappy software engineers.

DevOps to the rescue?

Wouldn't it be nice if the software engineer just worried about their application and its code, instead of all of the environments that it has to run on? In some places that is exactly what happens. Developers develop on their local laptops or in a development cloud and then check in their code and it moves to production. DevOps cleans up any problems with applications using single instance bottle-necked services, out of sync versions of centralized services, or adding load balancing services to the front end or back end of the application. The App developers have no clue what mess they have caused with their code changes, or a new version of service that they are using. Somehow we need to make sure that the application developer is still connected to the application architecture but disconnected  from the complexity of managing multiple environments.

Single Definition Multiple Environments

Working on my Local machine

One approach that I have been looking at is having the ability to define my application as a set of service templates. In this simple example I have a simple Node JS application that uses Redis and MongoDB. If I use a yaml format. It might look something like this.

MyApp:     
  Services:  
    web: NodeJS  
      ports: 80  
      links: mqueue, database  
    mqueue: Redis  
      ports: 6789  
    database: MongoDB  
      ports: 25678, 31502  

So with this definition I would like to deploy my application on my local box, using Virtual Box. I put this yaml file in my home directory of the application. This should be very familiar to those of you that have used docker-compose. Now I should be able to launch my application on my local machine using a command similar to docker-compose.

$ caade up

After a couple of minutes my multi-service application is running on my local laptop.

I can change the application code and even make changes to the services that I need to work with.

Working in a Development Cloud

Now that I have it running on my laptop I want to make sure that I can run it in a cloud. Most organizations work with development clouds. Typically development clouds are not as big as production and test clouds but give the developer a good place to try out new code and debug problems found in production and test environments. Ideally the developer should use the same application definition and just point to another environment to launch the application.

$ caade up --env=Dev

This launches the same application in the development environment. Which could be a OpenStack, VMWare or Kubernetes based SDI solution. The developer really does not care about how the infrastructure gets provisioned, just that it is done quickly and reliably. On quick inspection we see a slight difference in the services that are running in the development cloud. There is another instance of the NodeJS service running. This comes from the service definition of the NodeJS service. The NodeJS service is defined to have multiple instance in the development cloud and only one instance in the local environment.

NodeJS.yml - Service Definition

1. NodeJS:  
2.   Local:  
3.    web:  
4.       image: node-3.0.2  
5.       port: 1337  
6.   Dev:  
7.     web:  
8.      image: node-3.0.2  
9.      port: 1337  
10.     worker:  
11.      image: node-3.0.2  
12.      port:1338  
13.      cardinality: 3  
14.   Test: …  
15.   Prod: …  

This definition is produced by the service and stack developer not the application developer. So the service can be reused by several developers and can be defined for different environments (Local, Dev, Test, & Production). This ensures that services are defined for the different requirements of the environments. For example Production NodeJS might have a NGNX load balancer on the front end of it for serving up NodeJS web services for each user logged in. The key is that this is defined for the Service

that is reused. This increases reusability and quality at the same time.

Working in the Test Cloud

Now that I have tried my application in the development cloud. It is time to run it through a series of tests before it gets pushed to production. This is just as easy for the developer as working in the development cloud.

$ caade up --env=Test

$ caade run --env=Test --exec runTestSuites

We launched the environment and then run the test suites in that environment. When the environment launches you can see additional instances of the same services we have seen before in the development cloud. Additionally, there is a new service running in the environment. The Perf Monitor Service is also running. It is monitoring the performance of the services while the tests are running. Where did the definition of this service come from? It came from the application stack definition. This definition just like the service definition can show that the application can have a different service landscape for each environment. But the software developer still sees them as the same. That is to say, code should not change based on the environment that is running the application. This decouples the application from the environment and frees up the software developer to focus on code and not environments.

What about Production 

The ultimate goal of course is to get the application into production. Some organizations, the smart ones, don't let developers publish directly into production without some gates to pass thru. So instead of just calling "Caade up --env=Prod" we have a publish mechanism that versions the application, its configurations and supporting services.

$ caade publish --version=1.0.2

In this case the application is published and tagged with version 1.0.2. Once the application is published, it will then launch the environment if it is not currently running. If it is running then it will "upgrade the service" to the new version. The upgrade process will be covered in another blog. Needless to say it allows for rolling updates with minimum or no downtime. As you can see additional services have been added and some taken away from the test environment.

Happy "Coder" Happy Company

The software engineer in this story focuses on writing software not on the environment. Services are being reused from application to application. Environment requirements are being met with service and application definitions. Stack and service developers are focusing on writing services for reuse instead of fixing application developers code. Now your company can run fast and deploy quality products into production,a

Toolbox Full of Container Technology

This last weekend I started cleaning out my garage. I said I started, because my garage will never completely be cleaned out. But I did manage to clean up my tool box. I found lots of interesting things in my tool box. Some broken tools, some tools that I have no idea how they got there. I am sure my neighbors and friends probably are missing them.  I also found half started projects that just got thrown in their that don't matter anymore. Probably in a hurry to clean up the mess in the garage. Anyway. I started thinking about all of these tools in my tool box, and of course comparing them to the tools I use at work. They are not things I can touch like the hardware in my tool box, but they are invaluable tools that I can use to get jobs done. I quickly broke down the tools into categories and found a group of tools that I have downloaded, played with and am using everyday that fit in the "Container Technology" category. So I gathered those tools together and decided to write this blog to help me keep track of what container tool to use for what job. This is what I have so far. Please note this is not a complete list, but a good start I think.

Container Tools

• Docker - Container Definition and nice CLI to control Containers
• Docker Compose - Define applications that contain more than one container.
• Mesos - Meta-scheduler, Distribute Kernel
• Docker Swarm - Container Scheduler that can talk to MesosS
• Kubernetes - Container Scheduler that can schedule and control Containers.
• Marathon - Service based scheduler for containers. Plugin to Mesos
• Chronos - Temporal Based Scheduler for containers. Plugin to Mesos

This is how I see everything fitting together.

Docker (Production - 1.7.0)

This helps developers define how to configure a container. Starts with a base image and then allows you to run commands, expose ports, and copy data into the container through a simple configuration file. Docker also gives CLI and REST API to allow users to control the individual containers. It requires a Docker controller running on a physical or virtual machine.

Docker Compose (Beta - 1.7.0)

Gives developers the ability to define an application as a set of micro-services. It shows the dependencies between the micro-services (containers). Which ports they expose to each other, their startup order, and any shared resources(data) between the containers that make up the application.

Mesos (Production - 0.23.0)

"A distributed systems kernel" from the mesos web site.

Mesos describes itself as a distributed system kernel. It is responsible for picking physical compute nodes to run containers, jobs, micro-services, etc... I gets telemetry from the physical hardware and can determine which machines can best handle incoming jobs. It then provisions machines and executes job on those machines. Schedulers like Kubernetes, Marathon, Chronos, and Docker Swarm sit ontop of Mesos and act as the orchestrator for the containers running in the cloud.

Schedulers

Docker Swarm (Still in Alpha - 0.3.0 )

If you want to use a familiar Docker API you can use Docker Swarm with Mesos to control multiple containers on mutilple hosts. There is rumor it may also allow you to talk to Kubernetes in the future. Check out http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/ for more information.

Kubernetes ( pre-production Beta - 0.19.0 )

• If you want to launch groups of containers (K8 Pods) co-scheduled and co-located together, sharing resources on the same machine.
• If you want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
• if you want to use the K8s label-based service-discovery, load-balancing, and replication control. Very cool stuff for managing large number of containers.
• If you want to manage groups of containers(K8 Pods).
• If you want to load balance in isolation (K8 Pods).

Marathon (Production - 0.8.2)

• If you want to launch applications that contain long running heterogeneous apps/services (Docker and Non-Docker).
• If you want to use Mesos attributes for constraint-based scheduling.
• If you want to use application groups and dependencies to launch, scale, or upgrade related services.
• If you want to use event driven health checks to automatically restart unhealthy services.
• If you want to integrate HAProxy or Consul for service discovery.
• If you want a nice web UI or REST API to launch and monitor apps.

Chronos (Production - 2.3.4)

• If you want to launch applications that contain short running heterogeneous apps/services (Docker and Non-Docker).
• If you want to schedule a remporal task to run at a specific time/schedule, just like cron.
• If you want to schedule a DAG workflow of dependent tasks.
• If you want a nice web UI or REST API to launch and monitor apps.
• If  you want to use a scheduler that was built from the start with Mesos in mind.

Ok. This is the start. I am sure many of you have more to add to the list. Please let me know what I am missing.

DWP.