Introduction to systemd-nspawn Containers: chroot on Steroids

Containers are a technology that enables the isolation of applications through operating system-level virtualization. This allows for the running of multiple isolated instances of systems on a single host without the need for additional overhead associated with full virtual machine virtualization. In the Linux world, containers have gained significant popularity due to their efficiency and flexibility, allowing developers and system administrators to easily package, distribute, and manage applications in a consistent and portable environment.

Systemd-nspawn is a tool included in the systemd suite that is used to run lightweight system containers in the Linux environment. It is an integral part of the systemd init system, meaning it is available on most modern Linux distributions. This tool allows for the operation of isolated system spaces, each with its own set of processes and services, and even the ability to run different Linux distributions within a single physical host.

How does systemd-nspawn differ from other solutions like Docker or LXC?

Systemd-nspawn differs from Docker and LXC in several aspects:

• Level of Isolation: Systemd-nspawn provides isolation at the level of filesystem, network, and processes, but it is not as deeply integrated with Linux kernel functionalities as LXC or Docker, which offer more advanced features like cgroups for resource management.
• Simplicity: Systemd-nspawn is simpler to use for those already familiar with the systemd ecosystem. It does not require additional container management tools, which can be beneficial for smaller projects or users who prefer simplicity.
• Application: Unlike Docker, which is focused on applications as containers, systemd-nspawn is better suited for running complete operating systems for testing or development purposes.
• Integration with systemd: As a part of systemd, nspawn naturally integrates with its services, allowing easy management of containers as system services.

Systemd-nspawn is a container tool provided with systemd, used for creating and managing lightweight system containers on Linux platforms. This tool enables the operation of isolated system environments that share the same operating system kernel with the host but have their own independent user environment, processes, filesystem, and network configuration. Systemd-nspawn utilizes Linux kernel features like namespaces for spatial isolation and cgroups for resource control, thus offering significant isolation between containers and the host system.

Comparing systemd-nspawn to chroot – what are the additional capabilities?

Chroot is one of the oldest isolation mechanisms in Unix and Linux environments, which allows changing the root directory for a selected process and its descendants. It is a basic form of isolation, limited only to file access paths. In contrast, systemd-nspawn offers broader isolation and control over system resources:

• User space isolation: Includes not only the filesystem but also processes, users, sessions, and networks.
• Resource control: Integration with cgroups allows for controlling the use of CPU, memory, disk space, and other resources by each container.
• System service management: The ability to run and manage containers as systemd services, which facilitates automation and monitoring.

Example Configuration of a basic container with a base system

Assuming we want to create a simple systemd-nspawn container on a Linux system that isolates an Ubuntu environment, here are the configuration steps:

1. Install required packages and prepare the system

sudo apt install debootstrap systemd-container
sudo mkdir -p /var/lib/machines/

2. Initialize the container (for this purpose, I will use Ubuntu Jammy as the base system for the container)

sudo debootstrap jammy /var/lib/machines/developer

3. Set the root password for the container

sudo systemd-nspawn -D  /var/lib/machines/developer
passwd

4. Start the container

sudo systemd-nspawn -b -D /var/lib/machines/developer

Below, I include animations from the container startup:

Running a systemd-nspawn container as a system service

Running a systemd-nspawn container as a system service is another aspect of effective container management, enabling automation, monitoring, and easy lifecycle management of containers. Here is how you can configure and manage a systemd-nspawn container as a system service using systemd. The following command launches the container as a background service, including restarting with the system:

sudo systemctl enable --now [email protected]

To check the status of the running container service, use the command:

sudo systemctl status [email protected]

The container can be easily restarted or stopped using the commands:

sudo systemctl restart [email protected]
sudo systemctl stop [email protected]

Container activity logs are available through journalctl, which facilitates monitoring and diagnosing issues:

sudo journalctl -u [email protected]

SSH Login Configuration for systemd-nspawn containers

Offering SSH access to a systemd-nspawn container allows for various configuration methods that ensure secure and controlled user access management. Here are three options to consider to enable access to the container:

Option 1: Set up a chroot directory and configure sshd_config for SSH user isolation.
Advantages:

• Simple and easy setup.

Disadvantages:

• Need to create a new SSH user on the host.
• Services running in the chroot environment must be managed from the host.

Option 2: Set up a container using systemd-nspawn with SSH redirection.
Advantages:

• Services can be managed and run like regular systemd services within the container.
• The host can limit resources (RAM, CPU) assigned to the container.

Disadvantages:

• Requires additional steps when setting up the container (e.g., enabling auto-login).
• Both host and container must have the same user.
• scp and rsync do not work from the client side.

Option 3: Set up a container using systemd-nspawn with its own SSH server on a different port.
Advantages:

• Similar benefits to option 2, but scp/rsync works seamlessly.
• The user needs to be set up only on the container side.

Disadvantages:

• Requires additional steps in container configuration.

The choice of the right option depends on specific requirements and preferences for security management and resource availability. Later in the article, I will present option 2.

Option 2: Setting Up a Container Using systemd-nspawn with SSH Redirection

1. Create a user art on the host side

sudo adduser  art 

2. Create a user on the container side

sudo systemd-nspawn -b -D /var/lib/machines/developer
useradd -m -s /bin/bash art

3. Configure SSH redirection At the end of the /etc/ssh/sshd_config file, add

Match User art
    ForceCommand machinectl shell art@developer

4. Add polkit configuration
   Create a file /etc/polkit-1/localauthority/90-mandatory.d/99-machine.pkla with the content:

[Permit art to spawn shell in container]
Identity=unix-user:art
Action=org.freedesktop.machine1.shell
ResultAny=yes
ResultActive=yes
ResultInactive=yes

5. Restart the SSH server

sudo systemctl restart ssh

Example of successful login:

Import and Export of containers using systemd-nspawn

Managing containers with systemd-nspawn also includes the ability to import and export containers. These functions facilitate migration between systems and creating backups. Below, I describe how to perform the import and export of containers using systemd-nspawn.

Importing Containers

Importing a container involves bringing an existing container instance, which was previously exported or created on another system, into the local systemd-nspawn environment. Importing can be done using the machinectl tool, which is part of systemd and allows managing virtual machines and containers.

1. Prepare the container image: The image can be in RAW format, tarball (TAR), or another supported format. The image should contain a filesystem that meets the expectations of systemd-nspawn.
2. Using the machinectl command:

sudo machinectl import-tar <container_name>.tar

Exporting Containers

Exporting a container is the process of creating a portable image from an existing container, which can then be imported onto another system or kept as a backup. Exporting also uses machinectl.

sudo machinectl export-tar <conainer_name> path/to/the/image.tar

The export-tar command creates a tar archive containing the entire filesystem of the container. The image can then be transferred between systems or stored as a backup.

Practical aspects

Systemd-nspawn, as a container tool, can be applied in various environments and scenarios, ranging from development to production settings. Below, I will consider real-world applications of systemd-nspawn and compare its performance and resource management with other popular container solutions.

Real-world Applications of systemd-nspawn in production and development

1. Isolated Development Environments: Systemd-nspawn is an ideal tool for creating isolated development environments, where each developer can have their own separate environment with all necessary dependencies, without affecting the main operating system or the work of other developers. This approach minimizes dependency conflicts and facilitates application testing in environments close to production.

2. Testing and QA Environments: In Quality Assurance (QA) environments, systemd-nspawn can be used to quickly create and destroy containers for testing purposes, allowing easy replication of environments and isolation of changes. Containers can be configured with different software versions, simulating various operational conditions that may occur in production.

3. Production Deployments of Microservices: In a microservice architecture, where each microservice can be run in a separate container, systemd-nspawn can provide simplicity in management and isolation, while being less resource-intensive than full virtual machines.

Comparing performance and resource management in systemd-nspawn containers

1. Performance: Systemd-nspawn, by using less abstraction than traditional virtual machines, offers higher performance because containers share the host system’s kernel and do not require additional overhead associated with hardware emulation. Compared to other container tools, such as Docker or LXC, systemd-nspawn may provide comparable performance with simpler configuration, but with fewer features related to network and resource management.

2. Resource Management: Systemd-nspawn integrates with systemd on the host, allowing the use of cgroups to manage resources. Containers can be restricted in terms of CPU usage, memory, disk space, and network bandwidth. Compared to Docker, systemd-nspawn is less complex in configuration and management, though it may offer less granular control over resources and isolation. In production applications, where resource management and isolation are crucial, Docker may be more suitable due to its more advanced network management and security features. However, for smaller projects or where simplicity and lower overhead are more desirable, systemd-nspawn provides an effective alternative.

Summary

Systemd-nspawn, as a tool for managing system containers, offers several advantages that make it a valuable solution in various usage scenarios:

1. Simplicity and integration with systemd: As a tool integrated with systemd, systemd-nspawn easily cooperates with existing system configurations and services, enabling easy management of containers as system services.
2. Lightweight isolation: Systemd-nspawn provides effective isolation of operating systems without the significant overhead associated with hardware virtualization, making it faster than traditional virtual machines.
3. Flexibility in resource management: The ability to control resources such as CPU and RAM through cgroups allows for efficient management of container performance.

Possible issues and their solutions

Despite its many advantages, users of systemd-nspawn may also encounter certain challenges:

1. Limited networking functionality: Compared to more sophisticated container tools like Docker, systemd-nspawn offers fewer options in terms of advanced network configuration. A solution to this problem could be the use of additional network management tools such as iptables or firewalld for manual network configuration in containers.
2. Smaller community and support: Since systemd-nspawn is not as widely used as Docker, available resources, documentation, and community support may be less extensive. Active participation in Linux and systemd communities can help in finding solutions and support more easily.
3. Security management: Although systemd-nspawn provides a certain level of isolation, it is not as strong as the isolation offered by some other tools. Regular updates, proper configuration, and the application of security practices are key to minimizing risk.

Systemd-nspawn is a robust tool for those looking for a simple yet effective solution for isolating applications and environments in Linux. Its integration with systemd and low operational overhead make it an attractive choice, especially for production and development environments where resource management and simplicity are crucial.

References

1. https://www.docker.com/
2. https://kubernetes.io
3. https://wiki.archlinux.org/title/chroot
4. https://wiki.archlinux.org/title/Polkit
5. https://wiki.archlinux.org/title/systemd-nspawn
6. https://kilabit.info/journal/2022/chrooting_ssh_user_into_systemd-nspawn/