xzCompress files to the smallest size with xz
xz uses the LZMA2 algorithm to get the best compression of the common tools, while still decompressing quickly. It's used for release tarballs (.tar.xz), kernel and distribution images, and long-term archives; -T0 uses all cores.
xz [OPTION]... [FILE]... | unxz FILE.xz | xzcat FILE.xzYou're moving last month's logs to cold storage where every gigabyte costs, and want the smallest files.
xzLZMA2 compressor from XZ Utils, producing FILE.xz.
LZMA2 is an LZ77-style compressor with a huge dictionary (8 MiB at -6, 64 MiB at -9, versus gzip's 32 KiB), so it finds repeats far back in the data, plus range coding with context models that predict each bit from what came before. Finding long matches in a big dictionary is what makes compression slow and memory-hungry; decompression only follows the references and is fast. The .xz container stores the data in blocks with an index and a CRC64 per stream; with -T, xz cuts the input into independent blocks (three times the dictionary size each) so cores can work in parallel, at a slight cost in ratio. Decompression memory depends on the dictionary the file was made with, which is why xz has a memory limit (-M). In 2024 a backdoor was found in the liblzma library of XZ Utils 5.6.0 and 5.6.1 (CVE-2024-3094), which could hijack sshd through systemd's libsystemd; distributions rolled back before it reached stable releases.
-kKeep the original file-d / unxzDecompress-c / xzcatWrite to stdout-0 ... -9Preset; -6 default, -9 needs ~674 MiB to compress-eExtreme: slower search for a slightly better ratio-T NThreads for compression; -T0 = one per core-lList streams, blocks, sizes and ratio-tTest integrity-vVerbose, with progress-M LIMITMemory usage limit, e.g. -M 512MiBtar -cJfCreate a .tar.xz archivexz -9 on a small-memory container or CI runner-9 needs about 674 MiB to compress (and each thread more), which can get the job OOM-killed.
Stay at -6, or limit with -M 512MiBUsing xz for logs you grep every dayCompression is slow; for frequently written data the CPU cost outweighs the space saved.
gzip or zstd for hot data, xz for archivesExpecting xz -T0 output to be byte-identical to single-threaded outputMulti-threaded mode splits the input into blocks, which changes the file (and its checksum).
Pin -T1 (or a fixed -T N) when you need reproducible archivesTry these on a Linux machine of your own. Each task has a hint and a solution.
Compress a copy of a log from ~/logs with xz, keeping the original, then show its ratio.
Compress ~/data into data.tar.xz using all CPU cores.
Check which version of XZ Utils the sandbox has.
xzCompress files to the smallest size with xz
xz uses the LZMA2 algorithm to get the best compression of the common tools, while still decompressing quickly. It's used for release tarballs (.tar.xz), kernel and distribution images, and long-term archives; -T0 uses all cores.
xz [OPTION]... [FILE]... | unxz FILE.xz | xzcat FILE.xzYou're moving last month's logs to cold storage where every gigabyte costs, and want the smallest files.
xzLZMA2 compressor from XZ Utils, producing FILE.xz.
LZMA2 is an LZ77-style compressor with a huge dictionary (8 MiB at -6, 64 MiB at -9, versus gzip's 32 KiB), so it finds repeats far back in the data, plus range coding with context models that predict each bit from what came before. Finding long matches in a big dictionary is what makes compression slow and memory-hungry; decompression only follows the references and is fast. The .xz container stores the data in blocks with an index and a CRC64 per stream; with -T, xz cuts the input into independent blocks (three times the dictionary size each) so cores can work in parallel, at a slight cost in ratio. Decompression memory depends on the dictionary the file was made with, which is why xz has a memory limit (-M). In 2024 a backdoor was found in the liblzma library of XZ Utils 5.6.0 and 5.6.1 (CVE-2024-3094), which could hijack sshd through systemd's libsystemd; distributions rolled back before it reached stable releases.
-kKeep the original file-d / unxzDecompress-c / xzcatWrite to stdout-0 ... -9Preset; -6 default, -9 needs ~674 MiB to compress-eExtreme: slower search for a slightly better ratio-T NThreads for compression; -T0 = one per core-lList streams, blocks, sizes and ratio-tTest integrity-vVerbose, with progress-M LIMITMemory usage limit, e.g. -M 512MiBtar -cJfCreate a .tar.xz archivexz -9 on a small-memory container or CI runner-9 needs about 674 MiB to compress (and each thread more), which can get the job OOM-killed.
Stay at -6, or limit with -M 512MiBUsing xz for logs you grep every dayCompression is slow; for frequently written data the CPU cost outweighs the space saved.
gzip or zstd for hot data, xz for archivesExpecting xz -T0 output to be byte-identical to single-threaded outputMulti-threaded mode splits the input into blocks, which changes the file (and its checksum).
Pin -T1 (or a fixed -T N) when you need reproducible archivesTry these on a Linux machine of your own. Each task has a hint and a solution.
Compress a copy of a log from ~/logs with xz, keeping the original, then show its ratio.
Compress ~/data into data.tar.xz using all CPU cores.
Check which version of XZ Utils the sandbox has.