Linux and Docker real-time guide for Ubuntu Realtime-Kernel and PREEMPT_RT

Author: Tobit Flatscher (2021 - 2024)

Overview

This is guide explains how one can develop inside/deploy a Docker container running real-time capable code on a Linux operating system. As such it walks you through:

• Basics of real-time systems and an overview of different real-time Linux approaches
• Set-up of a real-time system, in particular activating the Ubuntu realtime-kernel or alternatively the installation of PREEMPT_RT supplying simple scripts for automatically re-compiling the kernel
• Possible optimizations of a real-time system in order to minimise the latency and benchmarking the real-time performance by means of cyclictest
• Making the Nvidia driver work with PREEMPT_RT
• Required settings for a real-time capable container with a PREEMPT_RT patched host system

This guide does not apply to Windows and Macintosh operating systems as it relies on a PREEMPT_RT-patched host system and Docker on Windows and Macintosh actually runs with a light-weight virtual machine in the middle.

For an introduction into development with Docker and Docker-Compose, how to set it up in Visual Studio Code and a guide for using it with graphic user interfaces and ROS workspaces for applications in robotics have a look at my other repository docker-for-robotics.

Use-cases

This can be useful for several different applications, in particular:

• Controlling real-time robotics hardware from Linux systems, e.g. over EtherCAT (by using EtherCAT masters such as SOEM or IgH) or setting up containers for other robotic components such as the Franka Emika Panda collaborative robotic arm
• Numerical control by turning your desktop or single-board computer into a SoftPLC e.g. with CodeSYS Control or using LinuxCNC
• Real-time programming in audio development such as with Juce

but may also apply to software developers developing any other form of real-time capable code.

1. Linux for real-time applications

There are different ways of turning a vanilla Linux system into a real-time capable operating system. As outlined in doc/RealTimeLinux.md PREEMPT_RT is likely the most future-proof possibility as it is about to be included into the mainline of Linux. As of Ubuntu 22.04 it is also available to Ubuntu users out of the box with Ubuntu Pro.

As a Docker shares the host kernel allowing a container to run real-time capable code is just a matter of starting it with the right permissions as long as the host kernel is real-time capable. The set-up of a real-time capable Docker with the Ubuntu realtime-kernel/PREEMPT_RT is quite straight forward. All you need is:

• A Ubuntu operating system with realtime-kernel or alternatively PREEMPT_RT-patched Linux host operating system
• An arbitrary Docker container launched with the correct options so that it can set real-time priorities from inside the container as well as options for reducing the network latency

1.1 Installing the realtime-kernel

There are several ways of applying a PREEMPT_RT patch. Since February 2023 and Ubuntu 22.04 an Ubuntu realtime-kernel, that can be installed with a few terminal commands, is available through Ubuntu Pro. For personal use you can register up to five machines for free while commercial customers will have to sign up for a subscription. I recommend installing the patch in this way as it is by far the easiest and most reliable. The guide doc/PreemptRt.md walks you through the registration and installation procedure.

Alternatively, I have also lined out the installation procedure either by compilation from source or from an existing Debian package in doc/PreemptRt.md. The same procedure can also be performed with the provided scripts src/install_debian_preemptrt and src/compile_kernel_preemptrt (see screenshots below).

Installation script for downloading existing Debian package	Installation script for compilation from source

1.2 Setting up real-time privileges

After having patched your system and restarted it, booting into the freshly installed kernel, you should already be good to go and be able to launch a real-time capable Docker. In case you do not intend to use root as the user inside the Docker you furthermore will have to have give yourself a name of a user that belongs to a group with real-time privileges on your host computer. How this can be done can be found in doc/PreemptRt.md.

1.3 Launching the Docker

After having successfully installed PREEMPT_RT, it is sufficient to execute the Docker with the options:

    cap_add:
      - SYS_NICE
    ulimits:
      rtprio: 99
      rttime: -1 # corresponds to 'unlimited'
      memlock: 8428281856
    network_mode: host

Then any process from inside the Docker can set real-time priorities rtprio (e.g. by calling ::pthread_setschedparam from inside the C/C++ code or by using chrt from the command line). Generally it is advisable to not set rtprio to 99 (refer to pull request #3 for more information).

2. Example

This Github repository comes with a simple example that can be used to try it out. Inside the Docker container a cyclictest is run to assess the real-time performance of the system. You can compare the outcome to running it on your local system. There should be virtually no difference between the two, as can be seen from the graphs below:

For launching the cyclictest open the Docker by typing

$ docker-compose -f docker/docker-compose.yml up

then browse the folder benchmark/ and run the command

$ ./mklatencyplot.bash

This should create a latency histogram by measuring the difference between a thread's intended wake-up time and its actual wake up time (resulting in an image plot.png similar to the one above). This measures any form of latency caused by hardware, firmware and operating system. For more information on this test refer to OSADL.