Run multiple docker daemons on the same host

Configure sudo

First, we will update the sudoers file by launching visudo to allow the users to use the docker command and force them to use the proper daemon. To do that, append the following lines at the end of the file:

USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *, ! /usr/bin/docker *--priviledged*, ! /usr/bin/docker *host*

You must replace USER (present two times) by the username of the user you want to allow (you must duplicate the line for each user you want to allow). This will allow USER to launch the docker command as root, without a password, while restricting the user to use its docker instance by talking to the proper socket. We also prevent the user to run a command that contains the priviledged option and host keyword. This is meant to avoid them disabling namespaces for security reasons. If we don't, they could run containers as true root on the machine. This would make our isolation less useful. If you don't care about that, you can simply use:

USER ALL=(root) NOPASSWD: /usr/bin/docker -H unix\:///var/run/docker-USER.sock *

Configure username spaces

Edit /etc/subuid and /etc/subgid so you have two lines per users which will look like (depending on your system, you may already have the second line):

USER:USERID:1
USER:1XXXXX:65536

The first line is used to map the user of id 1 in the container (ie root) to the id of a normal user outside. The second line, defines which uid will be used to map other users: the first number gives the first id and the second the number of ids. The ranges must not collide between users. Since this is still probably obscure, let me give an example:

• I have a user named jenselme. It has uid 1000 on the system.
• I have a user named julien. It has uid 1001 on the system.
• To enable mapping of root in the container to jenselme on the host for jenselme's containers, I add jenselme:1000:1 to /etc/subuid and /etc/subgid
• To enable mapping of other users in jenselme's containers, I add: jenselme:100000:65536. So users will be dispatched to host uids between 100000 and 165535. For instance, user with id 33 in a container will have id 100032 on the host (id mapping starts at 100000, hence 100032 and not 100033).
• To enable mapping of root in the container to julien on the host for julien's containers, I add julien:1001:1 to /etc/subuid and /etc/subgid
• To enable mapping of other users in julien's containers, I add: julien:165536:65536. So users will be dispatched to host uids between 165536 and 231071. Which are the 65536 ids avaiable after jenselme's ids range.

So my files will contain:

jenselme:1000:1
jenselme:100000:65536
julien:1001:1
julien:165536:65536

To learn more about this, you can read my article dedicated to this subject

Docker unit file

To do that, you first need to create a docker@.service file so you can select for which user you want to run docker. This file must be located under /etc/systemd/system/ and have this content:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker-containerd.service
Wants=docker-storage-setup.service
Requires=docker-containerd.service rhel-push-plugin.socket registries.service

[Service]
Type=notify
EnvironmentFile=/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/dockerd-current \
          --add-runtime oci=/usr/libexec/docker/docker-runc-current \
          --default-runtime=oci \
          --authorization-plugin=rhel-push-plugin \
          --containerd /run/containerd.sock \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --init-path=/usr/libexec/docker/docker-init-current \
          --seccomp-profile=/etc/docker/seccomp.json \
          --userns-remap %i \
          --host unix:///var/run/docker-%i.sock \
          --pidfile /var/run/docker-%i.pid \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
TasksMax=8192
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

I added three command line options to the standard service. The %i will be replace by what comes after the @ when we will run docker (eg USER with systemctl start docker@USER.service). Let me detail the new options:

• --userns-remap %i: enables the username space for the user. This way, each users won't be able to see/use the images and containers of each other: when usermapping is enabled, docker keeps everything (images, containers, networks, volume …) separate for each mapping.
• --host unix:///var/run/docker-%i.sock: change the name of the socket to listen to. If we don't do this, all our instances would try to listen to unix:///var/run/docker.sock which wouldn't work.
• --pidfile /var/run/docker-%i.pid: change the name of the PID file of the process. If we don't do this, all our instances would try to use /var/run/docker.pid which wouldn't work.

If you want to use a different docker configuration file for each instance, you can add --config-file /etc/docker/daemon-%i.json to the list of option. This will make docker use /etc/docker/daemon-USER.json as a config file when running docker@USER. Note: the file must exist and be valid JSON, otherwise, docker won't start.

Note: I have commented the part that differs from the standard docker.service on fedora. If you use a different distribution, the content of this file may differ but you should be able to adapt it from mine. If you have a question, please leave a comment.

Don't forget to make systemd take the new file into account with systemctl daemon-reload.

Usage

At this point, everything is functional, you can start your daemons with systemctl start docker@USER.service. To use a daemon, you need to specify to docker which socket to use with the -H option like this: sudo docker -H unix:///var/run/docker-USER.sock. To ease the process, you can create an alias like this in your .bashrc file (adapt to your shell):

alias docker="sudo docker -H unix:///var/run/docker-$(whoami).sock"

After opening a new shell or doing source ~/.bashrc, you can use the docker command directly, eg:

docker run --rm busybox ls

I suggest you play around with it on multiple accounts to see how all of this works before read the end of the article.

docker-compose

You can use docker-compose like this sudo docker-compose -H unix:///var/run/docker-$(whoami).sock if you add the line below to you sudoers:

USER ALL=(root) NOPASSWD: /usr/bin/docker-compose -H unix:///var/run/docker-USER.sock *

However, since docker compose does HTTP request to your docker socket, you cannot rely on sudo to prevent the user from launching priviledged or unamespaced containers. You will need an authorization plugin for that like HBM or Twistlock AuthZ Broker. Note: I have not tested these plugins.

Conclusion

Everything should be working and you should be able to run as many docker daemons as you need. I hope I was clear on this quite technical subject. If I wasn't or if you just want to leave a remark, please leave a comment below!