zstd
zstd is a fast, lossless compression algorithm and tool suite. zstd is available in the Linux kernel to support initramfs compression, compressed kernel modules, and compressed swap space.
zstd originated at Facebook.
Installation
Kernel
General setup --->
Kernel compression mode (ZSTD) --->
(X) ZSTD
[*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
[*] Support initial ramdisk/ramfs compressed using ZSTD
[*] Enable loadable module support --->
Module compression mode (ZSTD) --->
(X) ZSTD
Memory Management options --->
[*] Compressed cache for swap pages (EXPERIMENTAL)
Compressed cache for swap pages default compressor (zstd) --->
(X) zstd
Cryptographic API --->
{*} Zstd compression algorithm
File systems --->
<M> F2FS filesystem support
[*] ZSTD compression support
USE flags
USE flags for app-arch/zstd zstd fast compression library
+lzma
|
Support for LZMA compression algorithm |
lz4
|
Enable support for lz4 compression (as implemented in app-arch/lz4) |
static-libs
|
Build static versions of dynamic libraries as well |
test
|
Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently) |
zlib
|
Add support for zlib compression |
Emerge
root #
emerge --ask app-arch/zstd
Usage
Invocation
user $
zstd --help
Parallelization
zstd supports parallel compression in the main zstd utility which can be configured by environment variables or the -T parameter. It does not support parallel decompression in the main tool.
However, a contrib tool pzstd (which is installed alongside zstd) can both compress and decompress in parallel. It takes a different argument for the number of threads (-p). pzstd is not as widely used as zstd.
Configuration
zstd can be configured via environment variables: ZSTD_NBTHREADS sets the number of threads to use for compression (0 means "autodetect number available") and ZSTD_CLEVEL sets the default compression level.
Create the following file:
/etc/env.d/99zstd
# Use number of threads available for parallel compression
ZSTD_NBTHREADS="0"
Then run:
root #
env-update && . /etc/profile
Portage
To use zstd for Portage compression of both installed files and binary packages:
/etc/portage/make.conf
PORTAGE_COMPRESS="zstd"
BINPKG_COMPRESS="zstd"
It's possible for zstd to work harder at the cost of more resource usage and compression time - to achieve a smaller file:
/etc/portage/make.conf
PORTAGE_COMPRESS="zstd"
BINPKG_COMPRESS="zstd"
# BINPKG_COMPRESS_FLAGS_ZSTD flags:
# * -T0 (already the default but adding here so it's not lost)
# * -22: maximum compression level
# * --ultra: work harder
BINPKG_COMPRESS_FLAGS_ZSTD="-T0 -22 --ultra"
Note that Portage 3.0.38 or newer will attempt to compress binpkgs in parallel based on MAKEOPTS.
Dracut
The kernel must be compiled with support for zstd compression!
To use zstd for Dracut's compression of initramfs:
/etc/dracut.conf.d/compression.conf
compress="zstd"
Currently busybox's kmod doesn't support zstd-compressed modules, and if busybox module is installed in initramfs it ovewrites original kmod. See here
Thus the busybox module should be blacklisted in Dracut:
/etc/dracut.conf
omit_dracutmodules+=" busybox "
systemd
systemd has a Meson configuration option default-compression to allow choosing the preferred default compression algorithm.
To use zstd:
/etc/portage/env/sys-apps/systemd
MYMESONARGS="-Ddefault-compression=zstd"
Debug symbols
Valgrind does not yet support this.
Fangrui Song has championed and successfully implemented zstd debug info support in much of the toolchain, with it now being part of the ELF specification. zstd has some significant advantages over the default zlib, including compression ratio.
Portage support to make this truly configurable without a user patch is tracked as bug #906367.
Debug symbols can be compressed using zstd with newer versions of the toolchain:
- >=sys-devel/binutils-2.40[zstd]
- >=dev-libs/elfutils-0.189[zstd]
- >=dev-debug/gdb-13.1[zstd]
- >=sys-devel/llvm-16[zstd] if using lld or lldb, etc
Note that GCC doesn't need USE=zstd as it has a different meaning there.
Portage can be patched to pass --compress-debug-sections=zstd
to enable this:
/etc/portage/patches/sys-apps/portage/compressdebug-zstd.patch
Make Portage use zstd for debug info compression.
Make sure binutils[zstd], elfutils[zstd], and gdb[zstd] are emerged first!
Portage bug to make this properly configurable: https://bugs.gentoo.org/906367
--- a/bin/estrip
+++ b/bin/estrip
@@ -295,7 +295,7 @@ save_elf_debug() {
mv "${splitdebug}" "${dst}"
else
local objcopy_flags="--only-keep-debug"
- ${FEATURES_compressdebug} && objcopy_flags+=" --compress-debug-sections"
+ ${FEATURES_compressdebug} && objcopy_flags+=" --compress-debug-sections=zstd"
${OBJCOPY} ${objcopy_flags} "${src}" "${dst}" &&
${OBJCOPY} --add-gnu-debuglink="${dst}" "${src}"
fi
Removal
Unmerge
root #
emerge --ask --depclean --verbose app-arch/zstd
See also
- Gzip — a data compression utility maintained by the GNU project that uses Lempel-Ziv (LZ77) encoding
- Bzip2 — high-quality, patent free file compressor using the Burrows-Wheeler algorithm.
- Xz-utils — LZMA2-based data compression utility.