In my previous blog entry, I mentioned that Docker didn't provide any way to add host-to-ip mappings to
/etc/hosts. Well, the most recent version of Docker (1.3.1) now addresses that issue with a new add hosts option. Let's take a quick look at what it does.
Given a standard default
% docker run -it ubuntu cat /etc/hosts 172.17.0.8 b14031841b2b 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters
We can add a mapping for server 'foo' at '10.0.0.3'.
% docker run -it --add-host foo:10.0.0.3 ubuntu cat /etc/hosts 172.17.0.9 189a650112db ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet 10.0.0.3 foo
We can also add mappings for multiple servers, in this case 'foo' and 'bar'.
% docker run -it --add-host foo:10.0.0.3 --add-host bar:10.7.3.21 ubuntu cat /etc/hosts 172.17.0.13 3f8543ecc372 ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet 10.7.3.21 bar 10.0.0.3 foo
So far so good. But there are a couple more scenarios to consider. The first is how to handle the case where a single server is hosting multiple services. These services could be in a single Docker container, separate containers or not in a container at all. What's important is that they are both on some remote server.
So let's consider the case where our services are named
svcb. We could try to add both of them as host mappings. Let's see how that works out.
% docker run -it --add-host svca:10.0.0.9 --add-host svcb:10.0.0.9 ubuntu cat /etc/hosts 172.17.0.16 5d7eaa5dcd16 ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet 10.0.0.9 svcb 10.0.0.9 svca
Both services are pointing to the same IP address, but I'm not sure this is valid for
/etc/hosts. The format is supposed to be
IP hostname [hostalias]*.
Lucky for us, there is another option. When invoking
add-host, we can include both services inside of double quotes. It's not documented, but it does work as you can see below.
% docker run -it --add-host "svca svcb":10.0.0.9 ubuntu cat /etc/hosts 172.17.0.17 380df28b4612 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 10.0.0.9 svca svcb
There we go, both services pointing to the same IP address and
/etc/hosts written in a valid format.
Another issue is how to provide hostaliases of localhost. As far as I can tell, Docker still doesn't have anything to address this issue. For the time being this means that I'll still be using Mungehosts to do that work.
add-host is a valuable new tool in the Docker toolbox for pointing to remote services. It's worked well for all the cases I needed for it and it's easy to use. It's good to see Docker adding more config options to make our lives easier.
Update: The new Docker version 1.3.1 addresses some issues I describe in this post. Read about what problems Docker has solved.
Did you know that Docker doesn't let you (easily) modify
/etc/hosts? You can't programmatically add aliases for localhost or new hostname-to-ip mappings. In short, Docker treats
/etc/hosts like a static resource.
No problem. This will just take a quick Unix script to solve.
Attempt 1, the slate is wiped clean
No big deal, right? We simply write ourselves a little
sed script to alias localhost and add a couple additional hostname mappings. We can run the
sed script from our Dockerfile.
Well if you tried this at home, you'd quickly find out that it doesn't work. Each
RUN command in a Dockerfile does another commit to the image. Ordinarily, that's just a space problem but Docker treats
/etc/hosts special. It will overwrite it whenever it wants without any regard for your modifications. In a nutshell I wasn't able to persist any changes made during the Dockerfile.
Attempt 2, mount up
Okay, so we simply run our
sed script when we run our container. Docker run will execute our startup script, running our
sed script, updating
/etc/hosts and finally running our server.
sed: cannot rename /etc/sedl8ySxL: Device or resource busy
Unfortunately, that doesn't work either. The way
sed -i works is to read from the original file, write to a temp file and then move the temp file over the original file. This doesn't work in this case because
/etc/hosts is a mount point.
root@07734c969aa6:/usr/local/src# df -h Filesystem Size Used Avail Use% Mounted on rootfs 19G 3.9G 14G 23% / none 19G 3.9G 14G 23% / tmpfs 1005M 0 1005M 0% /dev shm 64M 0 64M 0% /dev/shm /dev/sda1 19G 3.9G 14G 23% /etc/hosts <---- none 1.1T 159G 881G 16% /code tmpfs 1005M 0 1005M 0% /proc/kcore
How are other people solving this problem today? Some people seem to be using
dnsmasq as a way of bypassing the problem of
/etc/hosts. Running another dns server process as a workaround seems excessive, but I'm not sure I blame them. Doing a search of issues in the Docker Github repo found 125 issues which mentioned
On Stackoverflow there is the realization that
/etc/hosts is now writable, but doesn't persist changes in running container.
Even so, I hadn't seen anyone provide a programmatic way to address the issue.
Attempt 3, Mungehosts is born
I took a step back and decided to write my own utility, Mungehosts. It solves my major pain points:
- Programmatic interface
- Can add aliases for localhost (ipv4 and ipv6)
- Can add hostname-to-ip mappings
- Works with Docker
- Compiled executable (Linux binary in releases)
- No dependencies except C runtime
- Small size (167 KB)
Check it out and provide feedback. I know that the smart Docker folks will eventually get all of the features into the main application. When that happens Mungehosts will retire to pasture, but it will be a good workaround until then.
I've been playing around with building Docker images and noticed I had a few that had
<none> as both the repository and tag info.
% docker images REPOSITORY TAG IMAGE ID ... <none> <none> c1e0e48a03bb <none> <none> c8bf15029bc0
I tried to delete these untagged images, but couldn't.
% docker rmi c1e0e48a03bb Error response from daemon: Conflict, cannot delete c1e0e48a03bb because the container 5f266c452a8b is using it, use -f to force
Ok, so we'll try to force deletion.
% docker rmi -f c1e0e48a03bb Error response from daemon: No such id: b5dc67806c5f1e4abfa079a1c99f6d782e5696a25a3c9beb16043fe2e7164d19 2014/10/18 10:04:27 Error: failed to remove one or more images
Mmm ... the original message said the image couldn't be deleted because it was in use by a container. I didn't have any running containers as I verified by doing
docker ps. But the issue is that docker holds on to images even from exited containers. Once you realize that, then this becomes much easier to resolve.
To delete all your exited containers.
% docker ps -a -q --filter "status=exited" | xargs docker rm
And to then delete all your untagged images.
% docker rmi `docker images -q --filter "dangling=true"`