Container Inception: Docker in nspawn Container on Linux

In the previous article, I introduced readers to the world of nspawn containers. In this part, I would like to expand on that topic by demonstrating how to run the Docker daemon in an nspawn container, using a specific configuration as an example. This may seem unusual, but in one of my projects, I faced a task that required such a solution. The server had Jenkins installed, responsible for building Docker images and running applications using docker-compose. Both Jenkins and Docker were running on the same machine instance, which posed a risk of system compromise, as users could manipulate the docker-compose.yaml file. To mitigate potential damage in case of an attack, one of the solutions I considered was using chroot (thanks for the suggestion, Jakub!).

However, chroot does not allow new system services to be run and is not considered a solid security solution. The answer turned out to be an nspawn container, which allowed me to separate the two services: Jenkins was placed in one container, while developer access and Docker were in another. Running Jenkins in an nspawn container is straightforward and does not differ from running it in a standard operating system. However, running the Docker daemon requires additional steps, which I have detailed in this article.

First, we need to prepare the environment and the nspawn container image that we intend to use. Let’s remember that the first step is to install the necessary system dependencies:

sudo debootstrap jammy /var/lib/machines/docker-container

The next crucial step in preparing the base container image is setting the password for the root user in the nspawn container. This can be done by temporarily starting the container using the following commands:

sudo systemd-nspawn -D  /var/lib/machines/docker-container
passwd

Compared to the command described in the previous article for starting the container, this time we need to add the --system-call-filter flag with the parameters add_key, bpf, and keyctl. This configuration grants the container access to the system calls add_key, bpf and keyctl.

Adding this configuration may still pose a security risk, although it is significantly lower than completely disabling user namespaces, as was necessary before the introduction of cgroups v2. Additionally, I included the --network-veth flag in the command, which enforces the use of virtual networking for the container.

sudo systemd-nspawn --system-call-filter='add_key bpf keyctl' --network-veth -b -D /var/lib/machines/docker-container

Now, the installation of Docker inside the nspawn container is the same as on a physical system. The only difference is the need to start the system service responsible for virtual networking at the beginning.

systemctl enable --now systemd-networkd systemd-resolved

apt-get install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update

apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

In Figure 1, I present a simple test of running the command docker run hello-world in an nspawn container.

Another issue I encountered was running the container as a system service. In the previous article, we used the method with systemd:

sudo systemctl enable --now [email protected]

Since our container requires a specific configuration, the above command will start the container, but the internal service, such as Docker, will stop working. To solve this problem, I copied the file [email protected] and modified its configuration:

sudo cp /usr/lib/systemd/system/[email protected] /usr/lib/systemd/system/docker-container.service

In it, I replaced the line:

ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i

with:

ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --system-call-filter='add_key bpf keyctl' --machine=docker-container

In the above line, I added the previously mentioned --system-call-filter flag and removed the -U flag, according to the description of this issue. Then, I executed the following commands:

systemctl daemon-reload
systemctl start docker-container

The last issue I would like to address in this article is the configuration of a reverse proxy, which allows the Docker service to be accessed on the local network. The configuration is very simple and uses the container name docker-container as the hostname. Since nspawn integrates well with the system, name resolution is done using container names. Figure 2 shows an example of using the ping command.

First, let’s set up the service inside our nspawn container:

echo “Hello this is nginx from docker in nspawn container” > index.html
docker run -v /root:/usr/share/nginx/html:ro -p 8080:80 nginx:stable-alpine-slim

Once we have our Docker service configured, we install the nginx package on the host system using the command:

sudo apt install nginx

and modify the /etc/nginx/sites-available/default file to:

server {
    listen [::]:80;
    listen 80;

    server_name _;

    location ~ (/.*) {
        proxy_pass http://docker-container:80;
    }
}

Finally, we restart the nginx service:

sudo systemctl restart nginx

References

1. https://www.docker.com/
2. https://wiki.archlinux.org/title/systemd-nspawn
3. https://mwalkowski.com/post/introduction-to-systemd-nspawn-containers-chroot-on-steroids/