Skip to content

Disks

Jump to bottom Edit New page
ArtMG edited this page Jan 25, 2023 · 14 revisions

A lot of the content in this is somewhat aged or underdeveloped. See also:

The persistent storage on Raspberry Pi is provided by the MicroSD slot. This is where the operating system, applications and most user data are stored. You can add additional storage devices via the USB ports.

Distributions such as NOOBS are loaded on to a FAT partition, but the install will convert to another partition format (type = Unknown?)

SD speed

The reader on the Raspberry Pi 2 is capable of speeds up to XXXXXXXXXXXXXX ???

Most SD cards are judged on sequential read/write speeds, like when you save or view images from a camera. However the usage profile of a PC OS disk is more random read/write, so this is what you really need to measure & score on.

For notes see https://www.raspberrypi.org/forums/viewtopic.php?p=53159

A lot of tests mention the Freeware Crystal Disk Mark which is compiled for Windows but may work under Wine?

For fuller details of how to test the read and/or write speed of media see https://github.com/artmg/lubuild/wiki/Disks-and-layout#Speed_and_Capacity_tests

Distro images

Most Distributions (distros) you are likely to install on your Raspberry Pi will come as a downloadable image file, which you can copy onto an SD card ready to set up and use.

These instructions assume you are on a debian-like linux operating system, such as Ubuntu, or even Raspbian on a Pi. If you use Windows or a Mac then please see (link TBC).

Beware: if you specify the wrong destination with the dd command you risk loosing data.

# find the device id of your SD card reader 
sudo fdisk -l

# supposing your SD card is mmcblk0 then use pne of the following...

# for an img.gz or img.zip file use 
gunzip -c path/to/imagefile.img.gz | sudo dd of=/dev/mmcblk0_CHECK_AND_REMOVE_ME

# or for a tar.gz file use
tar xzOf path/to/imagefile.tar.gz | sudo dd bs=1M of=/dev/mmcblk0_CHECK_AND_REMOVE_ME

# to check progress run this in another terminal
watch -n15 'sudo kill -USR1 $(pgrep ^dd)'

As you can see, you cannot just paste these in without checking the destination drive is correct and removing from that underscore to the end of the line. Remember if you dd of= to the wrong place you WILL loose your precious data.

mkusb, a safer alternative

The mkusb package is a handy wrapper to dd so you avoid toasting your data. It's readily available on ubuntu repos (see lubuild article) but is a little harder to get onto RaspberryPiOS.

# help https://help.ubuntu.com/community/mkusb/gui/tarball
wget https://github.com/sudodus/tarballs/raw/master/dus-plus.tar.xz
tar -xvf dus-plus.tar.xz
cd dus-tplus
sudo ./dus-installer i
# or whatever dependencies it asks for
sudo apt install -y pv xterm dialog zenity grub2-common

# then run it and follow the prompts
sudo -H dus ubuntu-XX.XX.iso

The grub2-common package was added in the hope of creating persistent (casper) volumes on the live USB, but that might not be working yet, so on Raspberry Pi maybe just stick to regular 'live-only' installers.

Expanding the filesystem

Most distributions come as a partition image which is sized just large enough for the files needed for the base installation. This does not leave room for any data, additional programs, or even the temporary files you will inevitably need when actually using the distro. You generally have two options:

Expand to the full SD card size

If you do nothing special yourself, this is what will happen by default. The distro installer's Expand Filesystem option will make the main filesystem take over all the space on the SD card.

Create an additional data partition

You may optionally wish to create a separate, secondary partition for data, to use up some of the SD card. Well, it's actually tertiary, as you have the boot and system partitions before it. The advantage to this approach is that your data will remain in place if ever you repeat the Copy Distro Image to SD card process, for example if you later do a fresh installation of a new upgraded version.

Use your Disk Partition management program (e.g. gnome-disk) to create a new parititon to occupy the rest of the SD card, from wherever you want the main filesystem to end (say half way through the SD card) all the way to the last sector.

Note that some distros come with multiple partitions, and therefore expect to be written to the disk (e.g. /dev/sdc) rather than to an individual parititon (e.g. /dev/sdc1). In these cases the upgrade procedure would make you loose your secondary data partiton and all its data. Still, as long as you have kept a copy of this data somewhere, you can always recreate the secondary data partition before re-doing the expand filesystem.

Additional persistent data partition after Pi OS

This section looks at various techniques to create, and maintain post re-build, a third partition. This will come after the two partitions normally transferred from an OS disk image, for the dual purposes:

  • limit the size of the second ext4 'Disk' partition that holds most system and home files - this is normally expanded during the build process and on a large disk can be wasteful
  • to hold data in a way that can persist the rebuild, instead of having to recopy such data manually after the rebuild.

Raspberry Pi OS images, like Raspbian before them, are disk images: a DOS/MBR boot record and partition table, pointing to two partitions containing filesystem images, all designed to be copied in one lump to a disk device (e.g. /dev/sdx rather than /dev/sdx9).

We need a technique to write the disk that will keep the data on the additional partition intact. Needless to say, any supposed 'persistence' should not be relied on for data integrity, and you should have a backup copy of any important data, in case the partition does become in any way corrupt during this process.

Possible techniques:

  • use (e.g.) fdisk to record the details of your third partition (start, end, type), then ensure that your disk image copy technique (e.g. dd) writes only what is needed into the to first data blocks on the disk. This technique relies on the disk image write completing before it reaches any part of the medium that might hold the signature and data of the third partition. It is unclear whether this would be reliable on an SSD, due to the difference between the supposed physical 'disk' layout and the places in flash where the data are actually stored.
  • mount the image as a loop and copy the filesystem images from within it
    • maybe, instead of dd the filesystem images, also create the filesystem and rsync copy the file and folder contents with permissions
    • perhaps refer to image-mount or this Q for any queries?
  • does LVM allow more flexibility with disks vs filesystems?
  • simply back up the data from your third partition, the recreate it and re-copy it once your build is complete. Although this is probably the least efficient in terms of disk writes, it is definitely simpler, more reliable, and unless this is an operation you need to carry out very frequently, it may be the most pragmatic.

Further reading:

As it happens, the first technique outlined above DOES work, providing that:

  • your disk image writing technique stops as soon as the image is finished
  • you expand your partition manually rather than relying on the auto-expansion in your distro setup

Some versions of Raspberry Pi OS, for example, won't support auto-expansion to a third partition limit. On Bullseye (2022-04-04), if you add a third partition you will need to power on once, wait for the 'cannot expand partition unless it's the last' error, then power off and restart to ignore the issue. Other dialogs popping up include Kbd Layout (or was this from reseting boot options?)

Alternative disk configurations

==Could== do with some comparative facts about USB speed vs SD speed of the Pi here - see what is in Other Devices # SSDs on Pi

As an alternative to storing the main OS on the SD card, you can use USB-connected devices. These might be USB thumb (flash) drives, USB Hard disk drives (USB HDD) or USB Solid State Drives (USB SSD).

Your options for alternate boot paths include:

  • ReadOnly distro with Write files on USB
  • BootOnly from SD with main Root on USB

For more info see:

  • step-by-step for BootOnly towards USB: [https://samhobbs.co.uk/2013/10/speed-up-your-pi-by-booting-to-a-usb-flash-drive]
  • the Pi's Boot Process explained: [https://thekandyancode.wordpress.com/2013/09/21/how-the-raspberry-pi-boots-up/]
  • automatically mounting external drives in Raspbian: [https://github.com/artmg/MuGammaPi/wiki/Rasbian-basics#Automount]

Headless setup

==Move this OUT== e.g. to Raspbian basics?

It may be that you can't (or don't want) to connect a screen, keyboard and mouse to your Pi when you set it up - this is known as a headless configuration. You will find some useful resources explaining the process in the following links:

The most important command I use to prep the SD image, as we are a multi-Pi-household is:

# choose a different computer name to help you identify your NEW headless Pi 
sudo nano /path/to/sd/card/etc/hostname

I'm fortunate that my home router will resolve host names into the newly assigned DHCP addresses, as long as I add the .lan suffix: e.g.

ssh [email protected]
Add a custom footer
Add a custom sidebar
Clone this wiki locally