Skip to content

Latest commit

 

History

History
1258 lines (968 loc) · 48.5 KB

uefi-linux-booting.md

File metadata and controls

1258 lines (968 loc) · 48.5 KB

A Linux administrator's tour of UEFI booting

Booting Linux from disk and common system administration tasks on a system with UEFI firmware


Objectives

  • Understand the linux boot process, including UEFI boot phases, loading the kernel, and spawning the init system.
  • Introduce various components of UEFI booting: where to find them on a linux system, and how to work with them.
  • Describe the phases of the UEFI boot process and how they relate to booting up a linux machine.
  • Perform common administration tasks associated with booting a linux machine.

Introduction

A firmware interface defines how the system initializes hardware, loads the operating system, and provides runtime services. UEFI (Unified Extensible Firmware Interface) is the standard firmware interface on most x86 machines, and it also supports ARM and some other architectures. It runs on laptops, desktops, servers, virtual machines, and more.

Understanding the UEFI boot process is important for troubleshooting boot issues. It also helps one to optimize system deployment and to ensure secure boot processes. UEFI encompasses more than just booting, but we will try to limit our discussion of it to that topic, focusing on how it relates to Linux system administration tasks.

This article examines the process of booting a linux system from a system administrator's point of view. It begins with the UEFI firmware and ends when the kernel spawns the init system. The legacy BIOS booting scheme, while useful for sysadmins to understand, is not covered here in detail. Plenty of good documentation exists for it, and we'd like to keep things relatively simple.

A brief look at some UEFI related objects on a Linux system

What are some UEFI related things you can see and work with on a Linux system? Later we'll cover these topics in more detail, but for now let's familiarize ourselves with some UEFI artifacts that are on a Linux system. Note that our demo system is running Debian 12, but we will point out differences with other Linux distributions when we encounter them.

Determining whether the running system is using UEFI

To check if a running system is booted in UEFI mode:

# ls -d /sys/firmware/efi
  • If the folder exists, the system is using UEFI.
  • If missing, the system is in Legacy (BIOS) mode.

UEFI NVRAM variables

UEFI uses NVRAM to store critical system settings, which include boot options, firmware configurations, hardware state information, and Secure Boot keys and settings.

Accessing UEFI NVRAM variables through the filesystem

How the kernel exposes UEFI NVRAM variables depends on the kernel version:

kernel version method
Before Linux 3.8 UEFI variables were managed using sysfs (/sys/firmware/efi/vars/).
Linux 3.8 (2013) Introduced efivarfs (/sys/firmware/efi/efivars/), which became the standard.

The files look the same to the user either way, but we will use the efivarfs interface in our examples. Each variable in UEFI NVRAM has a name, a type definition, and a binary value.

Enumerating UEFI NVRAM variables through the filesystem

To view the names of the variables in UEFI NVRAM, simply list the file names in /sys/firmware/efi/efivars/. The file names follow the naming convention of VariableName-VendorGUID. There is a special VendorGUID, called the EFI_GLOBAL_VARIABLE_GUID, which is used with a number of files in this directory, including BootOrder, individual boot options, and some variables related to Secure Boot, such as db and dbx. The db variable contains a list of allowed Secure Boot keys that can be used to authenticate bootloaders and operating system kernels, whereas the dbx variable contains a list of revoked keys. EFI_GLOBAL_VARIABLE_GUID has a constant value of 8be4df61-93ca-11d2-aa0d-00e098032b8c.

Enumerating global variables

To see the list of EFI global variables in nvram:

# ls /sys/firmware/efi/efivars/ | grep 8be4df61-93ca-11d2-aa0d-00e098032b8c | nl
     1	Boot0000-8be4df61-93ca-11d2-aa0d-00e098032b8c
     2	Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c
     3	Boot0002-8be4df61-93ca-11d2-aa0d-00e098032b8c
     4	Boot0003-8be4df61-93ca-11d2-aa0d-00e098032b8c
     5	BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
     6	BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
     7	BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c
     8	ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c
     9	ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
    10	ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
    11	ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
    12	dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
    13	dbtDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
    14	dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
    15	ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
    16	ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
    17	KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c
    18	KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
    19	OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c
    20	OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
    21	PK-8be4df61-93ca-11d2-aa0d-00e098032b8c
    22	PKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
    23	PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
    24	PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
    25	SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
    26	SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
    27	SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
    28	Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c
    29	VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c

We can see that there are 29 global variables in UEFI NVRAM on this system. How many variables total do we have on this system? Let's find out:

# ls /sys/firmware/efi/efivars/ | wc -l
69

Getting and setting values of UEFI NVRAM variables through the filesystem

As mentioned, the UEFI NVRAM variables are stored in binary files. To retrieve the value of a variable, simply read the contents of its corresponding file. To set a value, write to the file. As bash works primarily with text, working with these files is easier for tool writers using other languages, such as c. However, you can use tools like dd and xxd to read and write to these files from the shell. While it's both possible and sometimes useful or instructive to work directly with efivarfs, Linux also has a specialized tool for manipulating UEFI NVRAM variables, namely the efibootmgr command.

Introducing efibootmgr

To list the most common variables in UEFI NVRAM:

# efibootmgr
BootCurrent: 0000
Timeout: 2 seconds
BootOrder: 0000,0001,0002,0003
Boot0000* debian
Boot0001* UEFI:CD/DVD Drive
Boot0002* UEFI:Removable Device
Boot0003* UEFI:Network Device

Each line in the output represents a different variable. The purpose of each one in this list is mostly self explanatory. Of special note are boot options, which follow the naming convention Boot####, where #### is a hexadecimal number.

Variable Human-readable value Description
BootCurrent 0000 boot option that was used to start the current session
Timeout 2 seconds delay before automatically booting the default boot entry (0 means no delay)
BootOrder 0000,0001,0002,0003 priority in which to try boot options, moving to the next if the previous one fails
Boot0000 debian name of this boot option to present in a menu, for example

Let's look at boot option 0000 in detail:

# efibootmgr -v | grep Boot0000
Boot0000* debian	HD(1,GPT,a4beb095-37c9-417b-a83e-d95eb6d50df6,0x800,0x100000)/File(\EFI\DEBIAN\SHIMX64.EFI)

Explanation of output:

Field Meaning
Boot0000* UEFI boot entry number (* means it's active)
debian Label for this boot option
HD(1,...) The bootloader is on partition 1 (EFI System Partition)
GPT The disk uses GPT partitioning
a4beb095-... UUID of the partition (matches GPT table)
0x800 Start sector 2048 (Partition starts at 1MB)
0x100000 Partition size 1048576 sectors (512MB)
\EFI\DEBIAN\SHIMX64.EFI Bootloader path inside the EFI partition

GUID Partition Table

UEFI uses the GUID Partition Table (GPT), whereas the legacy (BIOS) method uses Master Boot Record (MBR). You can check whether a device uses MBR or GPT using the lsblk command:

# lsblk -o NAME,PTTYPE /dev/nvme0n1
NAME        PTTYPE
nvme0n1     gpt
├─nvme0n1p1 gpt
├─nvme0n1p2 gpt
└─nvme0n1p3 gpt
  • This displays the partition table type (dos for MBR, gpt for GPT).

EFI System Partition (ESP)

The EFI System Partition (ESP) is a small FAT32 partition used by UEFI firmware to store things like bootloaders, bootloader configuration files, and UEFI drivers for hardware initialization. It does not need a specific partition number or physical location on disk. The ESP is often mounted at /boot/efi/ on a Linux system, so that a bootloader such as grub can be easily updated. Note that when you dual boot Linux with another OS, both operating systems share the same ESP.

To find the EFI System Partition:

# fdisk -l | grep EFI
/dev/nvme0n1p1       2048    1050623    1048576   512M EFI System

Now let's run the blkid command on our ESP:

# blkid /dev/nvme0n1p1
/dev/nvme0n1p1: UUID="C20E-411A" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="a4beb095-37c9-417b-a83e-d95eb6d50df6"

We see that it has both a UUID and a PARTUUID. What is the difference?

The UUID is the unique identifier assigned to the filesystem on a partition. It is stored within the filesystem metadata (e.g., ext4, XFS, FAT32). This explains why the UUID of our ESP is not a UUID at all: instead of a UUID, FAT32 uses a Volume Serial Number.

PARTUUID is the unique identifier assigned to a partition by the GPT or MBR partition table. It is stored at the partition table level, not within the filesystem.

Now by using the lsblk command we can see if and where the ESP is mounted:

# lsblk /dev/nvme0n1p1
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme0n1p1 259:1    0  512M  0 part /boot/efi

Now let's take a peek at the files that debian installed on our ESP:

# find /boot/efi | grep -i debian
/boot/efi/EFI/debian
/boot/efi/EFI/debian/shimx64.efi  
/boot/efi/EFI/debian/grubx64.efi  
/boot/efi/EFI/debian/mmx64.efi	  
/boot/efi/EFI/debian/fbx64.efi	  
/boot/efi/EFI/debian/BOOTX64.CSV  
/boot/efi/EFI/debian/grub.cfg

We see that most of the file names on our EFI System Partition end with .efi. Note also that we have a grub config file in the debian subdirectory. Files with an .efi extension are binaries that the UEFI firmware can run. These are usually bootloaders but can also be utilities like the UEFI Shell. Let's digress for a moment to discuss the binary format of .efi files.

UEFI binaries

UEFI (Unified Extensible Firmware Interface) uses the PE (Portable Executable) format, which is the same executable format used by Windows. This is because UEFI evolved from Intel's EFI (Extensible Firmware Interface), which was developed with contributions from Microsoft.

PE is a standardized binary format used for executables, DLLs, and firmware applications. In the UEFI environment, PE binaries are used for bootloaders, UEFI shell applications, and even the OS kernel (if it supports direct booting).

How Linux works with PE in UEFI booting

The Linux kernel can boot directly from UEFI, without using a traditional bootloader such as grub. How is this possible, since Linux uses the ELF binary format and UEFI uses PE? A bit of sorcery is involved. If compiled with CONFIG_EFI_STUB=y, the Linux kernel embeds a PE-compatible EFI stub, making it loadable by UEFI. In the case of separate bootloaders such as grub, they are compiled as PE binaries (.efi files) to be recognized by UEFI firmware.

UEFI boot phases

Now that we're more familiar with the some of the components we'll be working with, let's back up for a moment to examine the overall boot process. The UEFI boot process consists of several distinct phases, each playing an important role in system initialization and OS booting. When booting a Linux machine with UEFI, these phases ensure proper hardware initialization, bootloader execution, and kernel handoff.

Briefly, these are the phases:

Phase Description
SEC (Security) Firmware integrity verification and CPU initialization.
PEI (Pre-EFI Initialization) Memory and chipset initialization.
DXE (Driver Execution Environment) Hardware initialization and firmware driver loading.
BDS (Boot Device Selection) Detects bootloaders in the EFI System Partition, loads the bootloader and verifies its signature.
TSL (Transient System Load) Executes the OS bootloader.
RT (RunTime) OS takes over, and UEFI Runtime Services remain available.

It's not necessary for a sysadmin to understand every phase in minute detail, but it helps to be familiar with them. Most of our management tasks occur in the Boot Device Selection and Transient System Load phases, so that is where we will continue our discussion.

Boot Device Selection

Steps of the Boot Device Selection phase

Here is a breakdown of the Boot Device Selection phase.

Boot Option Discovery

  • The firmware retrieves a list of boot options from non-volatile storage (NVRAM).
  • Each boot option represents a path to an EFI boot loader stored on a device.
  • The firmware may also dynamically add boot options by scanning devices (e.g., removable media, network PXE).

Boot Order Processing

  • The firmware follows the BootOrder variable, which lists boot options in priority order.
  • It attempts to load the first valid boot option in the sequence.

Boot Option Validation and Execution

  • If Secure Boot is enabled, the firmware verifies the digital signature of the EFI application.
  • If the boot option is valid, the firmware proceeds onto the TSL phase to execute the boot loader.

Fallback Boot Behavior

  • If no valid boot option succeeds, the firmware may:
    • Attempt the EFI default boot path (\EFI\Boot\BOOTx64.EFI on removable media).
    • Display a boot menu to the user.
    • Enter the UEFI Interactive Shell (if available).
    • Revert to legacy boot if Compatibility Support Module (CSM) is enabled.

Management tasks related to the Boot Device Selection phase

Preparing a disk for booting with UEFI

Here are the steps to initialize a disk as GPT, create an EFI System Partition, set the ESP flag, verify the partition table, and format the ESP partition with a FAT32 filesystem.

Create a GPT partition table:

# parted -s /dev/sdX mklabel gpt
  • Replace /dev/sdX with your actual disk (e.g., /dev/sdb).
  • This erases the existing partition table and creates a new GPT table.

Create an EFI System Partition (ESP):

# parted -s /dev/sdX mkpart ESP fat32 1MiB 512MiB
  • ESP is just a label for the partition. The name does not affect functionality, but ESP is commonly used as a label for an EFI System Partition.
  • fat32 is the filesystem type. parted does not actually format the partition, it only marks it as FAT32.
  • 1MiB is the start position (instead of 0). See Alignment Note
  • 512MiB is the end position. This means the total size of the partition is 511MiB.
  • An ESP partition of around 512MiB is standard for UEFI booting.

Set the ESP Flag:

# parted -s /dev/sdX set 1 esp on
  • This marks partition 1 as an EFI System Partition.

Protip: We could have created the partition table, created the ESP, and set the ESP flag with a single command:

# parted -s /dev/sdX mklabel gpt mkpart ESP fat32 1MiB 512MiB set 1 esp on

Display the partition layout to confirm the changes:

# parted -s /dev/sdX print

Format the ESP Partition as FAT32:

# mkfs.fat -F32 /dev/sdX1
  • Replace /dev/sdX1 with the actual partition (e.g., /dev/sdb1).

Final Check

To ensure everything is set up correctly:

# lsblk -o NAME,FSTYPE,SIZE,TYPE,MOUNTPOINT

Alignment Note:

It's generally a good practice to start partitions at 1 MiB (2048 sectors) instead of the very first sector, for the following reasons:

  1. Legacy BIOS and MBR Alignment issues.

    • In older MBR (Master Boot Record) partitioning, the first 512 bytes (sector 0) were reserved for the MBR and partition table.
    • Some older systems or tools might expect partitions to start at sector 63 or 2048 for alignment reasons.
    • GPT does not have this issue, but starting at 1 MiB (2048 sectors) ensures compatibility.
  2. Modern Disk Alignment (performance reasons).

    • Starting at 1 MiB ensures that partitions align with modern disk sector sizes, improving performance and compatibility.
    • Most modern SSDs and HDDs use 4K sector sizes (Advanced Format).
    • Many disks are optimized for 1 MiB alignment (2048 sectors).
    • If a partition starts at a non-optimal sector (like sector 34), it could result in misaligned I/O operations, reducing performance.
  3. GPT Reserved Space. Using 1 MiB (2048 sectors) ensures you do not overwrite the GPT header.

    • The first 34 sectors on a GPT disk are reserved for the Primary GPT Header and Partition Table.
    • If a partition starts before 34 sectors, it may overwrite this table, corrupting the partition structure.

Managing boot options

To add a boot option:

# efibootmgr --create --disk /dev/sdX --part 1 --label "Ubuntu" --loader "\EFI\ubuntu\grubx64.efi"  # using long options
# efibootmgr -c -d /dev/sdX -p 1 -L "MyLinux" -l "\EFI\MyLinux\grubx64.efi"                         # using short options
  • Replace /dev/sdX with the boot disk (e.g., /dev/sda).
  • Replace 1 with the EFI partition number.

To delete a boot option:

# efibootmgr --bootnum 0003 --delete-bootnum   # using long options
# efibootmgr -b 0003 -B                        # using short options

What happens if a UEFI boot option is missing or corrupt?

If the UEFI boot option is missing or incorrect:

  • The system may fail to boot and fall back to the UEFI firmware menu.
  • You may need to manually add the boot option using efibootmgr or the firmware interface, or by reinstalling the bootloader.
  • Some motherboards will attempt to boot from the default path (\EFI\BOOT\BOOTX64.EFI).

To apply what we've learned so far, we'll demonstrate how to create two boot options: one for the UEFI Shell, and one to boot Linux directly, without a traditional bootloader such as grub.

Installing a boot option to run the UEFI Shell

The UEFI Shell is a command-line environment that provides a pre-boot environment where you can execute commands, run scripts, and manage the system before an operating system loads. Many enterprise-class motherboards include the UEFI Shell as part of their firmware. Some examples of vendors are Dell, HP, Lenovo, and Supermicro. Most consumer PCs do not have the UEFI Shell preinstalled. As of this writing, you can download binaries of the UEFI Shell from github. Since our demo system is x86_64, we downloaded the file called shellx64.efi.

To install it, first identify the EFI System Partition (ESP):

# lsblk | grep efi
├─nvme0n1p1 259:1    0   512M  0 part /boot/efi

In our case, it's already mounted at /boot/efi. If it's not online then you should mount it after locating it:

fdisk -l | grep EFI
/dev/nvme0n1p1       2048    1050623    1048576   512M EFI System

Make a directory for the UEFI Shell on the EFI System Partition:

# mkdir /boot/efi/EFI/shell

Copy the binary to the ESP:

# cp shellx64.efi /boot/efi/EFI/shell/

Add the UEFI boot option:

sudo efibootmgr --create --disk /dev/nvme0n1 --part 1 --loader '\EFI\shell\shellx64.efi' --label "UEFI Shell"

Installing a boot option to run Linux without a separate bootloader

Here we demonstrate how to boot Linux directly from UEFI, without a traditional boot loader such as grub. As before, we begin by mounting the EFI System Partition:

# lsblk | grep efi
├─nvme0n1p1 259:1    0   512M  0 part /boot/efi

Again, it is already mounted at /boot/efi/.

Now we'll copy the kernel and initramfs to the ESP. Since our demo system is running debian, the name of the kernel begins with vmlinuz- and the name of the initramfs file begins with initrd.img-. To find our kernel and initramfs files:

# ls -1 /boot/{vmlinuz,initrd.img}-*
/boot/initrd.img-6.1.0-30-amd64
/boot/initrd.img-6.1.0-31-amd64
/boot/vmlinuz-6.1.0-30-amd64
/boot/vmlinuz-6.1.0-31-amd64

Copy the kernel and its corresponding initramfs file:

# mkdir /boot/efi/EFI/demo
# cp /boot/vmlinuz-6.1.0-31-amd64 /boot/efi/EFI/demo/vmlinuz.efi
# cp /boot/initrd.img-6.1.0-31-amd64 /boot/efi/EFI/demo/initrd.img

To boot directly into the Linux kernel, we need to create a boot option using efibootmgr.

First, find out which device the root filesystem is on:

# df -h | grep /$
/dev/nvme0n1p2  952G  130G  823G  14% /

Find the PARTUUID of the root partition:

# blkid /dev/nvme0n1p2
/dev/nvme0n1p2: UUID="6891b867-44c5-45bc-a913-1bc0d9fbb88e" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="649e329e-62d5-4407-be48-e81b59502139"
  • We want to take note of the PARTUUID, not the UUID.
  • UUID="6891b867-44c5-45bc-a913-1bc0d9fbb88e" : filesystem UUID (used for mounting, but not for boot configuration).
  • PARTUUID="649e329e-62d5-4407-be48-e81b59502139" : partition UUID (used by UEFI for boot management).

Then, create a UEFI boot option:

efibootmgr --create \
--disk /dev/nvme0n1 \
--part 1 \
--loader '\EFI\demo\vmlinuz.efi' \
--label 'direct linux demo' \
--unicode 'root=PARTUUID=649e329e-62d5-4407-be48-e81b59502139 initrd=\EFI\demo\initrd.img rw' \
--verbose
  • disk /dev/nvme0n1 specifies the disk containing the EFI System Partition (ESP)
  • part 1 specifies the partition number (ESP is usually 1)
  • loader '\EFI\demo\vmlinuz.efi' is the path to the EFI Linux kernel (vmlinuz.efi) inside the ESP
  • label 'direct linux demo' is the name for the boot option in the UEFI firmware
  • unicode 'root=PARTUUID=649e329e-62d5-4407-be48-e81b59502139 initrd=\EFI\demo\initrd.img rw' specifies kernel boot parameters
  • verbose shows detailed output

output:

BootCurrent: 0000
Timeout: 2 seconds
BootOrder: 0004,0000,0001,0002,0003
Boot0000* debian	HD(1,GPT,a4beb095-37c9-417b-a83e-d95eb6d50df6,0x800,0x100000)/File(\EFI\DEBIAN\SHIMX64.EFI)
Boot0001* UEFI:CD/DVD Drive	BBS(129,,0x0)
Boot0002* UEFI:Removable Device	BBS(130,,0x0)
Boot0003* UEFI:Network Device	BBS(131,,0x0)
Boot0004* direct linux demo	HD(1,GPT,a4beb095-37c9-417b-a83e-d95eb6d50df6,0x800,0x100000)/File(\EFI\demo\vmlinuz.efi)root=UUID=649e329e-62d5-4407-be48-e81b59502139 initrd=\EFI\demo\initrd.img rw
  • We now have a new boot option called "direct linux demo".
  • The BootOrder variable shows that "direct linux demo" will be the first attempted boot option.

Note: If Secure Boot is enabled, you need a signed kernel. You can use shim or manually sign the kernel with your own keys, which I will not cover here. Note that Secure Boot is disabled on our demo system.

Now, restart your system:

# reboot

If "direct linux demo" is not the first boot option listed in the BootOrder variable, enter the UEFI firmware settings or boot menu (F12, F2, Esc, or Del depending on your system) and select "direct linux demo".

PXE (Preboot Execution Environment) booting

The Boot Device Selection phase of the UEFI boot process can also boot Linux over a network using PXE. Network based Linux deployments, such as Linux Terminal Server Project or diskless workstations, rely on PXE to boot an OS. Configuring PXE is not difficult, but it's slightly outside our current scope since it requires setting up network services such as dhcp and tftp.

Managing the bootloader

Once we know how to manage boot entries, the next logical step is to learn how to manage the bootloader and its files. Since debian and RHEL based systems both typically use grub2 by default, we will use that bootloader for our examples. It's worth noting, however, that the locations of certain files and the commands to generate them are slightly different, depending on the distribution.

Feature Debian RHEL
EFI bootloader path /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/redhat/grubx64.efi
grub configuration file /boot/grub/grub.cfg /boot/grub2/grub.cfg
regenerating grub.cfg update-grub grub2-mkconfig -o /boot/grub2/grub.cfg

Here are some of the most common tasks relating to grub.

Checking grub configuration

You can view grub settings in the file: /etc/default/grub

Editing grub default boot option

Open /etc/default/grub in a text editor. Modify GRUB_DEFAULT to select a specific menu entry:

GRUB_DEFAULT=0  # First boot option

Regenerating grub.cfg

When should you regenerate the grub.cfg file?

You need to regenerate grub.cfg whenever there are changes that affect the boot configuration. Below are the most common scenarios.

After a Kernel Update

When a new kernel version is installed, grub.cfg needs to be updated so GRUB can recognize and boot it. If you don't regenerate it, GRUB may still boot an older kernel. Example: If linux-5.15 is installed, but you updated to linux-6.1, GRUB needs to know about it.

After Installing or Removing an OS

When dual-booting (e.g., adding Windows or another Linux distro), grub.cfg must be updated to detect the new OS. If an OS is removed, GRUB might still show it as an option, leading to boot errors. Example: You installed Windows alongside Linux, but GRUB does not show Windows.

After Changing GRUB Settings

If you edit /etc/default/grub (e.g., changing timeout, splash screen, kernel parameters), you must regenerate grub.cfg for the changes to apply. Example: You change GRUB_TIMEOUT=10 to GRUB_TIMEOUT=5 in /etc/default/grub.

After Changing Disk Partition Layout (e.g., Moving /boot)

If /boot or the EFI System Partition (ESP) is moved, resized, or restructured, grub.cfg must be updated. This ensures GRUB knows the correct partition for booting. Example: You moved /boot to a separate partition.

After reinstalling GRUB

If GRUB is corrupted or missing (e.g., after a failed update), reinstalling GRUB requires regenerating grub.cfg. Example: After running grub-install/grub2-install.

After Enabling or Disabling Secure Boot

If Secure Boot is enabled or disabled, the kernel and bootloader may need signing, affecting GRUB configuration. Certain distros use shimx64.efi to work with Secure Boot. Example: You disable Secure Boot and now need to boot an unsigned kernel.

If GRUB Fails to Detect a New Boot Device

If GRUB does not detect a new SSD, NVMe, or USB device as a boot option, updating grub.cfg may help. Example: You cloned your OS to a new SSD, but GRUB still points to the old drive.

Summary: When to Regenerate grub.cfg

  • Kernel update
  • Installing/removing an OS
  • Editing /etc/default/grub
  • Changing /boot/ or partition structure
  • Reinstalling GRUB
  • Secure Boot changes
  • New SSD/NVMe/USB boot device

Running grub-mkconfig

The grub-mkconfig command generates the grub.cfg file. The following steps occur when you run it:

Detects Installed Operating Systems
  • Scans for installed OSes (Linux, Windows, etc.) using os-prober (if available).
Finds Available Kernels
  • Checks /boot/ for kernel images and their initramfs.
Reads Configuration from /etc/default/grub
  • Uses the settings defined in /etc/default/grub (such as default kernel parameters).
Runs Scripts from /etc/grub.d/
  • Uses scripts in /etc/grub.d/ to generate different parts of the menu.
Creates or Updates grub.cfg
  • Writes the final bootloader configuration file.

The grub binary and configuration files are stored inside the EFI System Partition (ESP):

/boot/efi/EFI/debian/{grubx64.efi, grub.cfg}   # debian
/boot/efi/EFI/redhat/{grubx64.efi, grub.cfg}   # rhel

However, the main configuration is placed elsewhere first:

/boot/grub/grub.cfg    # debian
/boot/grub2/grub.cfg   # rhel

If you need to update grub.cfg, use:

# update-grub                              # debian
# grub2-mkconfig -o /boot/grub2/grub.cfg   # rhel

On a debian system, you could alternatively run:

grub-mkconfig -o /boot/grub/grub.cfg

Running grub-install

If your system has boot issues due to a missing or corrupted grub EFI bootloader, you can reinstall grub using the following steps.

Mount the EFI Partition (if not already mounted)

Ensure the EFI System Partition (ESP) is mounted:

# mount | grep efi

If not mounted, find it using:

# lsblk | grep efi

Then manually mount it (assuming ESP is /dev/sdX1):

# mount /dev/sdX1 /boot/efi

Reinstall grub EFI bootloader

Run the following command:

# grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian    # debian
# grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=redhat   # rhel

explanation of options:

  --target=x86_64-efi        # Specifies 64-bit UEFI grub.
  --efi-directory=/boot/efi  # Location of the EFI System Partition.
  --bootloader-id=debian     # Installs grub under /boot/efi/EFI/debian/.
  --bootloader-id=redhat     # Installs grub under /boot/efi/EFI/redhat/.

Regenerate the grub configuration

Run grub-mkconfig, as described in the previous section.

Verify the installation

Check whether the EFI grub files exist:

# ls /boot/efi/EFI/<vendor>    # <vendor> can be debian or redhat, for example

Reboot and test:

# reboot

We will now very briefly touch upon some issues relating to Secure Boot.

Managing Secure Boot

Secure Boot ensures only trusted OS bootloaders are executed. Linux works around this by having a shim (shim.efi) which is signed by Microsoft. This is a small boot loader which verifies and loads the main boot loader, such as grub. If an unsigned bootloader is used, Secure Boot will block execution unless a Machine Owner Key (MOK) is enrolled.

If you need out-of-the-box Secure Boot support, the best Linux distribution choices are, in no particular order: Ubuntu, Fedora, openSUSE, Debian, RHEL, AlmaLinux, Rocky Linux, and SUSE Linux Enterprise. For Arch-based or niche distributions, manual configuration is required. Secure Boot helps improve security, but in certain scenarios, disabling it might be an easier option.

To check Secure Boot status in Linux:

# mokutil --sb-state

Troubleshooting BDS-Related Boot Issues in Linux

Since the Boot Device Selection phase is responsible for selecting the Linux bootloader, failures here prevent the system from booting. Here are some common causes:

Issue Possible Cause Solution
No Bootable Device Found ESP is missing or corrupted Boot from a Linux live USB and repair the ESP. Also, ensure
the ESP is formatted as FAT32 and contains valid bootloader files.
Wrong OS boots Boot order is incorrect Use efibootmgr or go into UEFI settings to adjust the boot order.
Secure Boot blocks bootloader Bootloader is unsigned Disable Secure Boot or use shimx64.efi or use mokutil to enroll the MOK key.
Linux missing after Windows update Windows overwrote the UEFI boot option Use efibootmgr or boot into a Linux Live USB and reinstall grub.

Boot Device Selection summary

  • Scans boot devices for an EFI System Partition (ESP).
  • Selects a valid bootloader.
  • Verifies bootloader signatures if Secure Boot is enabled.
  • Supports network booting (PXE) for Linux deployment.
  • Can be managed using Linux tools (efibootmgr, mokutil, grub commands).

Transient System Load phase

Now that the bootloader has been located, loaded, and verified, it is time to run it. The Boot Device Selection phase is over, and the Transient System Load phase has begun. The TSL phase in the UEFI boot process is the point where the bootloader executes and transitions to the Linux kernel.

What Happens During the TSL Phase?

  • The UEFI firmware has already selected a bootable device during the BDS phase.
  • If Secure Boot is enabled, the signature of the EFI bootloader has already been checked.
  • The firmware starts the EFI bootloader.
  • The bootloader reads its configuration file from the ESP.
  • It provides a boot menu and options to load the kernel.
  • The bootloader locates the Linux kernel and initramfs.
  • The Linux kernel file (vmlinuz or bzImage) is copied into memory.
  • The initramfs (Initial RAM Filesystem) is loaded to handle essential hardware initialization. It contains essential drivers and files for booting.
  • The bootloader passes command line parameters to the kernel (e.g., root=/dev/sda1 ro quiet).
  • The bootloader exits, and the kernel begins.

What happens after the kernel takes control?

Before it can bring up the full system, the kernel needs to have the right drivers for things like storage devices (e.g., NVMe, SATA, RAID controllers) and filesystems (e.g., ext4, xfs, btrfs). To ensure this, it boots into the initramfs, which contains all the necessary modules and setup code. The process goes like this:

  • The kernel initializes system hardware and loads an initramfs.
  • When the kernel boots, it loads the initramfs image (if specified with initrd= in the bootloader).
  • The initramfs is unpacked into a RAM-based root filesystem (rootfs).
  • The kernel looks for an executable init, typically a shell script or a binary, to run as PID 1.
  • If it can't find init, the kernel panics.
  • The initramfs script loads necessary kernel modules.
  • If using LVM, RAID, or encrypted partitions (LUKS), the initramfs script sets them up (the kernel itself does not have built-in support to assemble these filesystems at boot).
  • The script identifies the real root filesystem based on: UUID (from /etc/fstab), kernel command line parameters (from the bootloader).
  • If encrypted, the user may be prompted for a passphrase to decrypt the root filesystem.
  • The kernel prepares to switch from initramfs to the real root filesystem by:
    • Mounting the real root filesystem.
    • Moving necessary files from initramfs.
    • Executing switch_root or pivot_root, replacing initramfs with the real root.
  • The kernel spawns the init process (/sbin/init, systemd or another init system like OpenRC, SysVinit).

Loading an initramfs during Linux boot is in theory optional but in practice usually required. Here’s when you need it and when you don’t:

When an initramfs is Required

  • Root Filesystem is on LVM, RAID, or Encrypted (LUKS) Partition

    • The kernel itself often does not have built-in support to assemble these filesystems at boot. The initramfs contains the necessary tools to mount them before switching to the root filesystem. Oftentimes, userspace tools are needed to set up these filesystems.
  • Kernel Lacks Built-in Drivers for Storage Devices

    • If the kernel does not have built-in drivers for the disk controller or filesystem (e.g., ext4, XFS), the initramfs provides these modules.
  • Custom Kernel Without Essential Features

    • If you build a custom kernel with minimal features, you may need an initramfs to supply necessary modules.
  • PXE Boot or Network Root Filesystem

    • The initramfs helps with setting up networking and fetching the root filesystem.

When an initramfs is Not Required

  • Monolithic Kernel with Built-in Support for Everything

    • If the kernel has all necessary drivers (e.g., for the disk controller and root filesystem), it can mount the root partition directly.
  • Simple Root Filesystem on a Standard Disk Partition

    • If your root filesystem is directly on a standard partition (e.g., /dev/sda1 formatted with ext4) and the kernel has built-in support for ext4, no initramfs is needed.
  • Embedded Systems with Minimal Boot Requirements

    • Some embedded Linux systems use a statically linked kernel with a fixed root filesystem.

How to Boot Without an Initramfs

  • If your system doesn't require it, you can configure the bootloader (GRUB, syslinux, etc.) to boot without it by removing initrd or initramfs entries.
  • When compiling your own kernel, you need to enable the required drivers (CONFIG_*) as built-in (=y) instead of modules (=m).

TSL related management tasks

View kernel parameters

On a running linux system, to view the command line parameters that the boot loader passed to the kernel:

# cat /proc/cmdline

List installed kernels

# dpkg -l | grep linux-image                # debian
# rpm -q kernel                             # rhel

Remove an old kernel

# apt remove linux-image-<version>          # debian
# yum remove kernel-<version>               # rhel

Reinstall the kernel

# apt reinstall linux-image-$(uname -r)     # debian
# yum reinstall kernel-$(uname -r)          # rhel

Managing initramfs/initrd

initramfs (initial RAM filesystem) and initrd (initial RAM disk) are temporary root filesystems used by the Linux kernel during boot before mounting the real root filesystem. They provide essential drivers, modules, and scripts required to initialize hardware and mount the root filesystem. Modern Linux distributions mostly use initramfs, which is a cpio archive, unlike initrd, which was a block device.

As with many things in the Linux world, there are two ways of managing initramfs, the debian way and the RHEL way. Here is a summary of their differences:

Feature Debian RHEL
Initramfs Management initramfs-tools dracut
Initramfs Location /boot/initrd.img-$(uname -r) /boot/initramfs-$(uname -r).img
View Initramfs Contents lsinitramfs /boot/initrd.img-$(uname -r) lsinitrd /boot/initramfs-$(uname -r).img
Rebuild Initramfs update-initramfs -u dracut --force

Checking existing initramfs/initrd

To list initramfs/initrd images that are installed on the system:

# ls -lh /boot/initrd.img-*                     # debian
# ls -lh /boot/initramfs-*                      # rhel

To view the contents of an initramfs image:

# lsinitramfs /boot/initrd.img-$(uname -r)      # debian
# lsinitrd /boot/initramfs-$(uname -r).img      # rhel

Updating and Regenerating initramfs

If a system fails to boot due to missing drivers, initramfs may lack necessary modules, for example for disk encryption or RAID setup. In this case, you may need to regenerate the initramfs.

Debian: Using update-initramfs

Regenerate initramfs for the current kernel:

# update-initramfs -u

Regenerate for a specific kernel version:

# update-initramfs -u -k <kernel_version>

Generate a completely new initramfs:

# update-initramfs -c -k <kernel_version>

Remove an initramfs:

# update-initramfs -d -k <kernel_version>
RHEL: Using dracut

Regenerate initramfs for the current kernel:

# dracut --force

Regenerate for a specific kernel version:

# dracut --force /boot/initramfs-<kernel_version>.img <kernel_version>

Generate a minimal initramfs:

# dracut --no-hostonly --force

Remove an initramfs:

# rm -f /boot/initramfs-<kernel_version>.img

Extracting and Inspecting initramfs/initrd

See what compression format the file uses:

file <initramfs file>

For example, on debian 12:

# file /boot/initrd.img-6.1.0-31-amd64
/boot/initrd.img-6.1.0-31-amd64: Zstandard compressed data (v0.8+), Dictionary ID: None
  • This file uses the zstd utility for compression.

And on Rocky 9:

# file  /boot/initramfs-5.14.0-503.23.1.el9_5.x86_64.img
/boot/initramfs-5.14.0-503.23.1.el9_5.x86_64.img: gzip compressed data, max compression, from Unix, original size modulo 2^32 78222336
  • This file uses the gzip utility for compression.

The uncompressed initramfs file is a cpio archive. Use the cpio command to view or extract its contents.

Adding Custom Modules to initramfs

To include custom kernel modules in the initramfs, you need to modify its configuration.

Debian
  • Edit the file: /etc/initramfs-tools/modules

  • Add the required module:

my_custom_module
  • Update initramfs:
# update-initramfs -u
RHEL
  • Create a config file: /etc/dracut.conf.d/custom.conf

  • Add:

add_drivers+=" my_custom_module "
  • Regenerate initramfs:
# dracut --force

Troubleshooting TSL-Related Boot Issues in Linux

As with the Boot Device Selection phase, failures during the Transient System Load phase will prevent the system from booting. Here are some common causes:

Issue Possible Cause Solution
grub loads but kernel is missing Kernel update broke boot configuration Boot into recovery mode and regenerate the grub config.
grub missing after Windows update Windows overwrote the UEFI bootloader Reinstall grub using a rescue USB stick.

After the TSL phase

We've already discussed managing the kernel binaries and initramfs images, since they're loaded during the TSL. At this point, the Linux kernel is running and the UEFI portion of the boot process is complete. However, we still have some ways to go before we have a usable system.

init

The init system is the first userspace process executed by the Linux kernel. Linux offers a few different init systems, such as systemd, SysVinit, OpenRC, and s6. However, systemd is the de facto standard, so we will limit our discussion of management tasks to that. An init system performs the following tasks:

  • Sets up and manages all other processes in the system.
  • Logs boot messages and service statuses for troubleshooting.
  • Ensures the proper mounting of filesystems (/, /home, /var, ...).
  • Boots into the correct state, such as multi-user mode, graphical mode, or recovery mode.
  • Starts essential services (networking, logging, ssh, etc.).
  • Monitors system processes and restarts failed services when necessary.
  • Allows enabling, disabling, starting, stopping, and restarting services.
  • Interacts with tools like udev to handle dynamic device management.
  • Controls system shutdown, reboot, and hibernation.

Once the init system is more or less finished bringing up userland, the user sees a login prompt (tty or GUI login manager like gdm, SDDM, LightDM). The user can log in and use the system.

Managing systemd

A full discussion of systemd is beyond the scope of this article, but here are some common things you can do.

Analyze boot time:

systemd-analyze

Identify slow startup services:

systemd-analyze blame

Check boot process:

systemctl list-units --failed

View logs from the current boot:

journalctl -b

View logs from the previous boot:

journalctl -b -1

View the current boot target:

systemctl get-default

Set the system to boot into CLI mode:

systemctl set-default multi-user.target

Set the system to boot into GUI mode:

systemctl set-default graphical.target

Reboot into rescue mode:

systemctl rescue

You can also boot into rescue mode by accessing the grub menu, highlighting the kernel entry you wish to boot, and appending this to the kernel line:

systemd.unit=rescue.target
  • Press Ctrl+X or F10 to boot.

Reboot into emergency mode:

systemctl emergency

To boot into emergency mode from the grub menu, follow the same steps as for rescue mode but instead append:

systemd.unit=emergency.target

Difference between systemd rescue mode and emergency mode

Both rescue mode and emergency mode are minimal recovery environments in systemd, but they serve different purposes and provide different levels of access for troubleshooting a Linux system.

Rescue Mode (rescue.target)

  • Purpose:

    • Used for basic system recovery when the system is partially functional but needs maintenance.
    • A minimal single-user mode with essential system services and mounted filesystems.
    • Allows administrators to fix broken configurations, reset passwords, and troubleshoot issues without a full multi-user environment.
  • Features:

    • Loads basic system services (sysinit.target).
    • Mounts all local filesystems (/, /usr, /var, etc.).
    • Does not start networking or graphical UI.
    • Prompts for the root user password.
  • When to Use Rescue Mode?

    • Fix misconfigured system services.
    • Reset forgotten passwords.
    • Repair broken configuration files.
    • Troubleshoot boot failures with access to system logs.

Emergency Mode (emergency.target)

  • Purpose:

    • A lower-level recovery mode for severe system issues where normal booting is impossible.
    • Used when the root filesystem is corrupted, unmounted, or inaccessible.
  • Features:

    • Loads the absolute minimum system services.
    • Root filesystem is not mounted (or only mounted in read-only mode).
    • No networking, multi-user mode, or systemd services.
    • Drops directly into a root shell (sh) for manual system repairs.
  • When to Use Emergency Mode?

    • Fix corrupt filesystems (fsck on /).
    • Manually mount partitions (mount -o remount,rw /).
    • Repair broken fstab entries that prevent booting.
    • Recover from missing or misconfigured system files.

Which Mode Should You Use?

  • If the system boots but needs repairs, use Rescue Mode (systemctl rescue).
  • If the system fails to boot completely, or / is corrupt/unavailable, use Emergency Mode (systemctl emergency).

Both modes require root access but Emergency Mode is more restrictive, requiring manual intervention to mount filesystems.

Single user mode example: Resetting the root password on a system with SELinux

This isn't a booting or init system issue per se, but it's an example of when you might want to interrupt the boot process to perform a system management task. In this case, the kernel spawns a shell as its first process instead of systemd.

Reboot into the system, adding the following kernel parameters: init=/bin/bash enforcing=0

Remount all necessary filesystems:

# mount -o remount,rw /

Reset the password:

# passwd root

Since SELinux is enabled, any password change won't take effect unless the security context is restored. Create a flag file to trigger SELinux relabeling:

# touch /.autorelabel

Reboot the system:

# exec /sbin/reboot -f

On reboot, SELinux will relabel the filesystem. The process may take a few minutes. Once the system comes back up, log in as root using the new password.