Development containers

Posted on 2024-01-02 in Programmation

I recently discovered the dev containers standard (or development containers for long) recently after trying to contribute to a project which had them enabled by default. It seems to be a new standard way to work with containers in development.

The goal is to provide a container you can use as a full featured environment for development. Your editor and its terminal are connected to the container and you run everything in it. So instead of running some things locally or wrapping everything in docker compose exec to run them on the container, you run everything directly in the container thanks to your editor. With VSCode, you can even personalize the container to install your favorite shell and theme (in my case, I use ZSH or Fish with starship), use your customized git configuration as well as any extra tools you might need.

It is compatible with plain docker as well as docker compose and supports VSCode and Pycharm (currently in beta in Pycharm). According to my tests, it works very well with VSCode and will be picked up automatically if you have the proper extensions installed (see the getting started section of the doc which is really good) and is still rocky in Pycharm (it even broke for a while after a Pycharm update). Once it supported correctly, it means no matter what editor you use (provided it supports the standard) you will have the same experience and behaviors. And for Pycharm, it means you can use it inside a container without the usual slowness regarding container creation for every command you launch.

I think it’s a very good idea and that we should use it more. It does come with a few issues if you use user namespaces with Docker like me. Let’s dig deeper for that use case.

User namespaces

The goal of user namespaces is to map a user inside the container to another user outside. I’ve explored this extensively in this article. For instance, I map the root user inside the container to my user outside. This way, all files created by root in the container will belong to me outside. It makes sharing files on volumes much easier on Linux where there is no abstraction layer to do UID conversions automatically. It also brings extra security: root inside the container is a mere user on the host. If you don’t run the container with the root user, files will belong to a weird UID outside the container.

Let’s illustrate this with an example. Let’s take this Dockerfile:

FROM debian:latest

RUN apt-get update && \
    apt-get install -y passwd

RUN groupadd --gid 1000 dev-user \
    && useradd --uid 1000 --gid dev-user --shell /bin/bash --create-home dev-user

With user namespacing enabled in the Docker daemon, build the image with docker build -f Dockerfile --tag test:latest . then run it with docker run -it --rm test:latest

If you look at permissions in the home folder or /usr/bin/bash you should see something like this:

root@435ec3552812:/# ls -alhn /home/
total 0
drwxr-xr-x 1    0    0  16 Dec 24 13:24 .
drwxr-xr-x 1    0    0 174 Dec 24 13:24 ..
drwxr-xr-x 1 1000 1000  54 Dec 24 13:24 dev-user
root@435ec3552812:/# ls -alh /home/
total 0
drwxr-xr-x 1 root     root      16 Dec 24 13:24 .
drwxr-xr-x 1 root     root     174 Dec 24 13:24 ..
drwxr-xr-x 1 dev-user dev-user  54 Dec 24 13:24 dev-user

Now, rerun with: docker run -it --rm --userns host test:latest to disable user namespacing for this run. You should see something like this:

root@565d6ae60c50:/# ls -alh /home/
total 0
drwxr-xr-x 1 dev-user  dev-user   16 Dec 24 13:24 .
drwxr-xr-x 1 dev-user  dev-user  174 Dec 24 13:26 ..
drwxr-xr-x 1 100000999 100000999  54 Dec 24 13:24 dev-user
root@565d6ae60c50:/# ls -alhn /home/
total 0
drwxr-xr-x 1      1000      1000  16 Dec 24 13:24 .
drwxr-xr-x 1      1000      1000 174 Dec 24 13:26 ..
drwxr-xr-x 1 100000999 100000999  54 Dec 24 13:24 dev-user

Root became user with UID 1000 (including for already present files like /usr/bin/bash) while user became something non existent (the remapped UID used during the build). Before, inside the container everything was correct including for files like /usr/bin/bash which was correctly owned by root with UID 0 inside the container.

Disabling user namespacing

Let’s now disable user namespacing in the Docker daemon and retry: docker run -it --rm test:latest. You should get something like this:

root@ed2f0d20f197:/# ls -aln /usr/bin/bash
-rwxr-xr-x 1 0 0 1265648 Apr 23  2023 /usr/bin/bash
root@ed2f0d20f197:/# ls -alh /usr/bin/bash
-rwxr-xr-x 1 root root 1.3M Apr 23  2023 /usr/bin/bash
root@ed2f0d20f197:/# ls -alh /home/
total 0
drwxr-xr-x 1 root     root     16 Dec 24 13:30 .
drwxr-xr-x 1 root     root      0 Dec 24 13:31 ..
drwxr-xr-x 1 dev-user dev-user 54 Dec 24 13:30 dev-user
root@ed2f0d20f197:/# ls -alhn /home/
total 0
drwxr-xr-x 1    0    0 16 Dec 24 13:30 .
drwxr-xr-x 1    0    0  0 Dec 24 13:31 ..
drwxr-xr-x 1 1000 1000 54 Dec 24 13:30 dev-user

So everything is fine. But, files created with root in the container will be owned by root outside. Files created in the container will be owned by whatever user has UID 1000 on your system (most likely you).

Problems with VSCode and devcontainers

By default, VSCode uses the root user inside the container: you will perform all your operations as root whether user namespacing is enabled or not. With user namespacing on, it’s not a big deal since everything will be mapped to the correct user. Without it, everything belongs to root which is a pain. You are also doing all your actions as root which is still weird and probably a bad idea since it may not be realistic to enforce user namespacing to all members of your team (without even thinking about open source project).

You can setup VSCode to use a different user. That will work fine without user namespacing. But with it, you UID outside the container will be whatever UID user namespacing attributed to you. So you won’t have read access to the files (unless you open read access for everybody but that’s just another bad idea). So you will be stuck and won’t be able to develop.

You could ask VScode to run the containers with the --userns host option (it’s also doable with docker compose by adding userns_mode: host to your service definition). But, since the user will be added during the build step and you can’t build your image with user ns disabled (or if it’s possible, I haven’t found how), the files of the newly added users won’t belong to it as we’ve seen earlier and VSCode won’t be able to install its server.

It turns out, VSCode has a nice feature: updateRemoteUserUID. It’s on by default and will change the UID of the user of the container (unless it’s root which must stay with UID 0) to the UID of your user. So, you can transparently create file inside or outside the container they will have the correct UID! Sadly (and despite what the doc is saying) it doesn’t seem to work for GID which will still be whatever it is in the container. But it’s still a huge step forward.

Here is the docker file I used (inside a .devcontainer folder):

FROM debian:latest

RUN apt-get update && \
    apt-get install -y passwd git

# Let’s use something that proably doesn’t exit on the system.
RUN groupadd --gid 5000 dev-user \
    && useradd --uid 5000 --gid dev-user --shell /bin/bash --create-home dev-user

RUN mkdir -p /app

USER dev-user

And the devcontainer.json (also inside the .devcontainer folder):

{
  "build": { "dockerfile": "Dockerfile" },
  "workspaceMount": "source=${localWorkspaceFolder},target=/app,type=bind",
  "workspaceFolder": "/app",
  "updateRemoteUserUID": true
}

Let’s recap:

  • Enabling user namespacing: besides using root as a user for all your actions, I don’t think it’s usable. And it’s probably a bad idea (you wouldn’t do it on your host system after all) and for your team mates that don’t have user namespacing enabled it will create a security issue since they are also running everything as root on the host.
  • Disabling user namespacing and using a dedicated user: thanks to VSCode and its updateRemoteUserUID feature everything will work correctly out of the box. I still think it’s better to explicitly set the UID and GID to 1000 when building the container: it’s probably the right value. Since the GID isn’t updated, it’s a very good default and will hide the internals away for most users.

Wrapping up

I’ve been a huge proponent of user namespacing for years since it makes everything go more smoothly on Linux while increasing security. It doesn’t seem to work well with dev containers. Since the spec allows you to remap the UID of a standard user inside the container to your UID outside, it’s not a very big deal.

I think from now on, I’ll only enable user namespacing if I really need it and launch all my dev works in containers as a standard user letting my IDE doing the UID remapping if needed.

I also think I’d leave user namespacing on in my production environment so that if an attacker gains root access in a container, they don’t have root access on the host.

And you, what do you think? Do you know more about devcontainers? If so, please leave a comment below.