Archives and compression
xz
Compress files to .xz with LZMA2 for high ratios, or decompress them with -d, unxz or xzcat.
xz [OPTION...] [FILE...]
xz usually produces the smallest files of the common compressors, at the price of slow
compression and a lot of memory at high levels. Decompression is reasonably fast, which is why
it suits files compressed once and downloaded many times: source tarballs (.tar.xz), kernel
images, distribution packages.
It follows the gzip conventions: xz FILE replaces the file with FILE.xz, -d or unxz
decompresses, -k keeps the input, -c writes to standard output, and xzcat prints the
contents. Levels run from -0 to -9 (default -6), and -e ("extreme") squeezes a little
more at any level. -T0 compresses with one thread per core.
When speed matters more than the last few percent, zstd is usually the better choice.
Examples
Compress a file, keeping the original
xz -k access.log
ls -l access.log*
-rw-rw-r-- 1 dev dev 2309 Oct 24 21:55 access.log -rw-rw-r-- 1 dev dev 624 Oct 24 21:55 access.log.xz
-k keeps access.log. Without it xz deletes the original once access.log.xz is
written.
Show sizes and ratio
Setup run before this example
xz -k access.logxz -l access.log.xz
Strms Blocks Compressed Uncompressed Ratio Check Filename
1 1 624 B 2309 B 0.270 CRC64 access.log.xz -l reads the .xz index: one stream, one block, 624 bytes compressed from 2,309. The
ratio column is compressed divided by original, so smaller is better, the opposite of
gzip's "space saved" figure. CRC64 is the integrity check stored with the data.
Watch the result per file
xz -v data/sales.csv
data/sales.csv: 228 B / 405 B = 0.563
-v prints the sizes and ratio when the file is done (on a large file it also shows
progress while working). The 405-byte CSV became 228 bytes, 0.563 of its size, and was
replaced by data/sales.csv.xz.
Decompress with -d or unxz
Setup run before this example
xz todo.txtunxz todo.txt.xz
ls todo*
todo.txt
unxz is xz -d. The .xz file is replaced by the original.
Read a compressed file without unpacking it
Setup run before this example
xz access.logxzcat access.log.xz | awk '{print $9}' | sort | uniq -c
11 200
2 401
2 403
4 404
1 500 Choose a preset level, or extreme mode
xz -0 -c access.log | wc -c
xz -9 -c access.log | wc -c
xz -9e -c access.log | wc -c
660 624 624
-0 is fastest and -9 compresses hardest; -e tries extra strategies at the same level.
On this tiny file -9 saves 36 bytes over -0 and -e finds nothing more. On large inputs -9 needs about 674 MiB of memory
to compress, so the default -6 is usually the sensible choice.
Use every CPU core
xz -T0 -k access.log
xz -t access.log.xz && echo valid
valid
-T0 starts one worker per core. Multithreaded mode splits the input into blocks, which
only helps on files of many megabytes, but it never hurts. -t then checks the result.
Make and unpack a .tar.xz
tar -cJf src.tar.xz src
mkdir out && tar -xf src.tar.xz -C out
ls out/src
Makefile app.py lib main.c util.sh
-J makes tar compress with xz. Extraction needs no flag: GNU tar recognizes the
format.
Pass xz options through tar
XZ_OPT='-9 -T0' tar -cJf data.tar.xz data
xz -l data.tar.xz
Strms Blocks Compressed Uncompressed Ratio Check Filename
1 1 1388 B 20.0 KiB 0.068 CRC64 data.tar.xz tar runs xz with its default settings. The XZ_OPT environment variable adds options to
every xz run, which is the simplest way to set a level for a .tar.xz. The uncompressed
size is the tar stream, padded to 20 KiB, which is why the ratio looks so good.
Decompressing over an existing file
Setup run before this example
xz -k notes.txtunxz notes.txt.xz
unxz: notes.txt: File exists
notes.txt already exists, so unxz stops with File exists and status 1. -f overwrites;
-c writes elsewhere.
Common pitfalls
xz filedeletesfilewhen it finishes. Use-kto keep it.High levels need a lot of memory:
-9uses roughly 674 MiB to compress and 65 MiB to decompress. On small machines stick to the default or lower.The xz ratio column (
xz -l) is compressed/original, so 0.270 means the file shrank to 27%; gzip's-lshows the opposite, the percentage saved..lzmais the older format from the same project.xzcan read it (xz --format=lzma -dorunlzma), but new files should be.xz.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
macOS does not ship the
xzcommand (Homebrew or MacPorts provide it), although its bsdtar can extract.tar.xz. On Linux and FreeBSD the options are the same XZ Utils ones.
Options worth knowing
A selection, described in our own words. For the full list, see xz(1) or xz --help.
| Option | What it does |
|---|---|
-d, --decompress | Decompress. |
-z, --compress | Compress, even when invoked as unxz. |
-k, --keep | Keep the input files. |
-f, --force | Overwrite existing files and compress links and already-compressed names. |
-c, --stdout | Write to standard output and keep the input. |
-t, --test | Verify the integrity of compressed files. |
-l, --list | Show information about .xz files: sizes, ratio, check type. |
-T, --threads N | Use N worker threads; 0 means one per CPU core. |
-e, --extreme | Try harder at the chosen level for a slightly smaller result. |
-0 | Fastest preset level (levels -0 to -9; default -6). |
-9 | Strongest preset level, with the highest memory use. |
-v, --verbose | Print progress and the final ratio; repeat for more detail. |
-q, --quiet | Suppress warnings. |
-F, --format FORMAT | Read or write FORMAT: auto, xz, lzma or raw. |
-C, --check CHECK | Integrity check to store: none, crc32, crc64 or sha256. |
-M, --memlimit LIMIT | Cap memory use, for example 512MiB or 50%. |
Related
gzipCompress files to .gz (replacing the original by default), or decompress them with -d or gunzip.bzip2Compress files to .bz2 with the Burrows-Wheeler algorithm, or decompress them with -d, bunzip2 or bzcat.zstdCompress files to .zst with Zstandard, fast at any ratio; decompress with -d, unzstd or zstdcat.tarBundle files and directories into one archive, list it, or extract it; optionally compressed with gzip, bzip2, xz or zstd.zcatPrint the decompressed contents of gzip files to standard output, leaving the files compressed.