User:Maffblaster/Infrastructure/Disk images

From Gentoo Wiki
Jump to:navigation Jump to:search

When working with small systems (such as SoCs, older slower systems, or embedded systems), it is useful to know how to manipulate disk images. This article will attempt to flesh out different creation and capture methods, mounting/attaching, conversion to run via virtual machines. This can be useful when attempting routine maintenance, package updates, etc into a disk image.

When a Linux-based operating system contained within the disk image is the same as the host computer architecture, it is simple to chroot into the disk image. In cases where the host computer architecture is different, then a qemu can be used to emulate the target system architecture and provide a static chroot.

Images

Create

Depending on the software which will used for operations, disk images can be created using a variety of tools.

Simple tooling

It is important to note when using disk image creation methods for the raw format type, the full size of the created file will be written to disk.

dd

dd, which is included as part of the @system set, will be present for all Gentoo supported computer architectures. It is possible to create a disk image of the raw format using this tool in a multitude of ways. A simple method that does not require mathematics is to use the count= and seek= parameters to specify the scope of the file to be created.

For a 20 GB file:

user $dd if=/dev/zero of=disk_image.img bs=1 count=0 seek=20G
fallocate

fallocate, which is included as part of the @system set, will be present for Gentoo supported computer architectures. Raw disk images can be quickly created using this tool:

user $fallocate --length 20GiB disk_image.img

Specialty tooling

Specialty tooling for manipulating disk images generally associated with and provided by virtual machine software suites such as Qemu or VirtualBox.

qemu

When working with qemu-img (from app-emulation/qemu), a 20 GiB raw disk image can be created via:

user $qemu-img create disk_image.img 20G

An advantage of working with qemu-img is that it can directly create disk images in format types specialized for virtual machines. For example, to create an image in qcow2 format, which is qemu's own disk image format type, issue:

user $qemu-img create -f qcow2 disk_image.img.qcow2 20G
Formatting 'disk_image.img.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16

Notice that qcow2 will utilize zlib compression by default, which is a space-saving advantage when compared to raw disk images. Note that using a non-raw disk image format will require conversion back to raw format before it will be bootable from a physical storage medium (SD card, SSD, etc.).

Partition, format, mount, and chroot

To partition and format the disk image use a standard partitioning utility such as parted or fdisk. Parted can be used specify filesystems during the partitioning phase, whereas fdisk cannot.

Once partitioning has been finished, follow the mount steps below using losetup to expose the partitions contained within the disk image to the host system. The partitions may need initial formatting using the appropriate mkfs utility, then mounted using the mount command before a chroot or qemu-user chroot will be possible.

Inspection

Using the file tool (from sys-apps/file) may provide information to indicate that the target file is a disk image, but fails to provide additional details about the image.

user $file disk_image.img
disk_image.img: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 41943039 sectors, extended partition table (last)

qemu-img

Disk image format can be determined using qemu-img:

user $qemu-img info disk_image.img
image: disk_image.img
file format: raw
virtual size: 20 GiB (21474836480 bytes)
disk size: 4 KiB
Child node '/file':
    filename: myimage.img
    protocol type: file
    file length: 20 GiB (21474836480 bytes)
    disk size: 4 KiB

Capture

There are many tools available in Gentoo to capture disk images. Capturing an image is nothing more than making backup of the system. Generally this involves physically connecting the disk to a system and creating a file from the disk image. Starting from easier and faster methods, and moving to harder and slower methods:

  1. Connect the disk by inserting it into an external USB device, then connecting the USB device to a system.
  2. Connect the disk directly to the system's motherboard.
  3. Connect the disk to another system, and then send the data to a new system

See the Backup article for more information on creating backups in general.

Once the disk is connected choose a tool to use to capture the disk image. For this example, a variant of dd will be used to ensure the integrity of the backup.

Note
It is important to verify the initial capture completes with perfect success to not encounter corruption issues.
Important
When capturing raw disk images, the source disk will likely need to be smaller than the destination disk. For example, if the source of the backup is 500 GB, the destination will need at least 500 GB free space in order for the disk image to be captured successfully. There are some exceptions to the free space rule depending on the tool used for capture. Some tools will compress the disk image as it is captured, which can save much space in the long run, however it is likely the capture will have to be decompressed in order to be accessed in a useful manner.

ddrescue

In the following example, ddrescue is used to capture one disk as a disk image.

root #ddrescue -f -n /dev/sdb /root/sdb_rescue.img /root/sdb_rescue.log

If all goes well, the captured disk image will be available at /root/sdb_rescue.img.

Mount

After the disk image has been captured, it is typical to mount the image as loop devices.

Discovery

It is wise to first check to see what loop devices are presently in use:

root #losetup --list

The following command can be used to attach to either a single or multi-partition disk image to a loop device:

root #losetup --partscan --find sda1.img

Listing the loop devices again should show something like the following:

root #losetup --list
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop0         0      0         0  1 /home/larry/sda1.img                  0     512

With the image attached to a loop device, is it now possible to run fsck commands on the image, or mount the image at a mount point.

View available partitions in the image by querying the loop device:

root #ls /dev/loop0p?
/dev/loop0p1  /dev/loop0p2

Filesystem types for each partition can be checked or resized using either a disk partitioning utility such as fdisk, parted, or gparted on the loop device as a whole.

root #fdisk --list /dev/loop0
Disk /dev/loop0: 100 MiB, 104857600 bytes, 204800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd414f95e

Device       Boot  Start    End Sectors Size Id Type
/dev/loop0p1       40960 143359  102400  50M  c W95 FAT32 (LBA)
/dev/loop0p2      143360 204799   61440  30M 83 Linux
root #parted /dev/loop0 print
Model: Loopback device (loopback)
Disk /dev/loop0: 105MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      21.0MB  73.4MB  52.4MB  primary  fat16        lba
 2      73.4MB  105MB   31.5MB  primary  ext4

Opening gparted for an easy online resize:

root #gparted -a /dev/loop0
GParted 1.4.0
configuration --enable-online-resize
libparted 3.5
btrfs images

Presuming the sda1.img file is a btrfs filesystem, the image can be checked or repaired just like any other btrfs device. See the Troubleshooting section below on repairing dirty filesystems.

Mount the image:

root #mount -o loop -t btrfs /home/larry/sda1.img /mnt/image

Unmount

Unmounting the image is typically no different any other unmount. In a rare instance, it may be necessary to lazily unmount an image when there's heavy disk I/O.

root #umount /mnt/image

Detach

If the image was attached to a loop device, it is proper to detach when finished to free up the loop device for other uses:

root #losetup --detach /dev/loop0

Convert

Converting raw disk image to VM disk image

Converting the raw image (.img) file to qcow2 format (requires app-emulation/qemu):

root #qemu-img convert -f raw -O qcow2 vm_sdd.img vm_sdd.img.qcow2

Trim

Resizing a partition inside a disk image file does not decrease the allocated, on disk size of the disk image file. However, after shrinking the size of the partition(s) inside the disk image, the end of the (uncompressed) file can be trimmed off to save space. Compression can be used to save an additional amount of space. This can be performed by determining the last sector number for the last partition in the disk image, then cutting off anything preceding the last used sector.

Warning
Any data that exists after the last sector will be irrevocably lost. Proceed with caution.

The command to do so will look something like the following:

user $truncate --size=$[(LAST_SECTOR+1)*BLOCK_SIZE] disk.img

Disk sector size can be determined using fdisk or sfdisk:

user $fdisk -l disk.img
user $sfdisk -l disk.img

Compress

To get the best backup-to-space ratio, compressing raw images after the capture will be highly beneficial.

Disk images can be compressed just like any other file. For example, to compress with an "ultra" high compression (--ultra) using 16 threads (-T16) with the zstd compression algorithm (requires app-arch/zstd):

user $zstd --ultra -22 -T16 /path/to/disk.img
disk.img            : 12.50%   (  59.6 GiB =>   7.46 GiB, disk.img.zst)

The example above displays a 59.6 GiB disk image compressing down to less than 8 GiBs. Successful compression levels may vary depending on the how much free space is in the disk image, among other variables.

Restore

Warning
Observe an extra level of precaution when restoring disk images to a destination block device (such as /dev/mmcblk, /dev/nvme0n, or /dev/sd). The write operation is destructive and one way only, since it removes the target's boot record, partition tables, and filesystem data. Without a disk image backup, the data on the block device will be lost forever.

As long as the block device which will be used for the restoration is large enough to hold the disk image, restoration is possible by writing the disk image directly to the attached block device. This can be performed a number of ways, such as using a dd implementation, like ddrescue:

root #ddrescue -f -n /root/disk.img /dev/sdb /root/disk.img.log

Compressed disk images can be restored to a attached block device by decompressing the file and writing the output directly to the block device.

For example, the file disk.img.xz was compressed using the xz utility (from app-arch/xz-utils). xzcat can be used to decompress the file directly onto a new block device:

xz decompression (with progress):

root #xzcat --verbose disk.img.xz > /dev/sdb

zstd decompression (with progress):

root #zstdcat --progress disk.img.zstd > /dev/sdb

This has the advantage of not needing to decompress the compressed disk image, which may require more space than the host file system has available.

Troubleshooting

Filesystem image will not mount

Filesystems occasionally have problems. For example, presume in the example below that sda1.img is a btrfs formatted filesystem image and is already attached to /dev/loop0.

root #mount -o loop -t btrfs /dev/loop0 /mnt/image
mount: /mnt/image: can't read superblock on /dev/loop0

Oh, no! Looks like there's a problem with the filesystem image. No to worry, this may be correctable. Filesystem images can be checked just like actual devices.

fat

ext4

btrfs

If the image is just one large btrfs filesystem, the following repair command should work.

root #btrfsck --repair /dev/loop0
enabling repair mode
WARNING:

	Do not use --repair unless you are advised to do so by a developer
	or an experienced user, and then only after having accepted that no
	fsck can successfully repair all types of filesystem corruption. Eg.
	some software or hardware bugs can fatally damage a volume.
	The operation will start in 10 seconds.
	Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/loop0
UUID: 3f345b95-ff66-46b2-b7f5-4a55d8961ea8
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
reset isize for dir 792 root 5
reset isize for dir 3426 root 5
reset isize for dir 4895 root 5
reset isize for dir 4940 root 5
reset isize for dir 5561 root 5
reset isize for dir 5571 root 5
reset isize for dir 258862 root 5
Trying to rebuild inode:9223372036855356061
Can't get file name for inode 9223372036855356061, using '9223372036855356061' as fallback
Can't get file type for inode 9223372036855356061, using FILE as fallback
Moving file '9223372036855356061' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 9223372036855356061
reset nbytes for ino 9223372036855356061 root 5
reset isize for dir 792 root 819
reset isize for dir 3426 root 819
reset isize for dir 4895 root 819
reset isize for dir 4940 root 819
reset isize for dir 5561 root 819
reset isize for dir 5571 root 819
reset isize for dir 258862 root 819
Trying to rebuild inode:9223372036855356061
Can't get file name for inode 9223372036855356061, using '9223372036855356061' as fallback
Can't get file type for inode 9223372036855356061, using FILE as fallback
Moving file '9223372036855356061' to 'lost+found' dir since it has no valid backref
Fixed the nlink of inode 9223372036855356061
reset nbytes for ino 9223372036855356061 root 819
T[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 173687640064 bytes used, no error found
total csum bytes: 151170848
total tree bytes: 2374696960
total fs tree bytes: 1938522112
total extent tree bytes: 230211584
btree space waste bytes: 532258492
file data blocks allocated: 3031879557120
 referenced 258574393344

Now that the filesystem has been repaired, proceed to mount as normal:

root #mount -o loop -t btrfs /home/larry/sda1.img /mnt/image

Yay! The issue has been fixed!

See also

  • qemu — a generic, open source hardware emulator and virtualization suite.