xz [option… ] [file… ]
unxz --decompress. unlzma --format=lzma --decompress.
xzcat --decompress --stdout. lzcat --format=lzma --decompress --stdout.
General-purpose data compression tool with command line syntax similar to gzip and bzip2.
LZMA Utils and raw compressed streams with no container format headers are also supported.
Without file reads from standard input and writes to standard output.
o When compressing, .xz or .lzma is appended to the source filename
o When decompressing, suffix is removed, recognizes .txz and .tlz, and replaces them with the .tar suffix.
file is not a regular file. Symbolic links are not followed
file has more than one hard link.
file has setuid, setgid, or sticky bit set.
SIGINFO or SIGUSR1 outputs progress information to standard error.
--verbose will display an updating progress indicator.
--memlimit=limit. or XZ_DEFAULTS=--memlimit=150MiB. --format=raw or --no-adjust)
in steps that don't match the compression level presets, e.g. if the limit is only slightly less than the
amount required for xz -9, the settings will be scaled down but not all the way down to xz -8.
.xz files can be concatenated as is. xz will decompress such files as if they were a single .xz file.
Padding (nulls, a multiple of four bytes) can be inserted between the concatenated parts or after the last part. . useful if the file is stored on a medium that allocates file sizes in 512-byte blocks.
Concatenation and padding are not allowed with max indicates the maximum value supported by the option.-z | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Version
xz --robot --version will print the version number of xz and liblzma in the following format:
XZ_VERSION=XYYYZZZS
LIBLZMA_VERSION=XYYYZZZS
X Major version.
YYY Minor version. Even numbers are stable. Odd numbers are alpha or beta versions.
ZZZ Patch level for stable releases or just a counter for development releases.
S Stability. 0 is alpha, 1 is beta, and 2 is stable. S should be always 2 when YYY is even.
XYYYZZZS are the same on both lines if xz and liblzma are from the same XZ Utils release.
Examples: 4.999.9beta is 49990091 and 5.0.0 is 50000002.
Memory limit information
xz --robot --info-memory prints a single line with three tab-separated columns:
1. Total amount of physical memory (RAM) in bytes
2. Memory usage limit for compression in bytes. A special value of zero indicates the default setting, which for single-
threaded mode is the same as no limit.
3. Memory usage limit for decompression in bytes. A special value of zero indicates the default setting, which for single-
threaded mode is the same as no limit.
In the future, the output of xz --robot --info-memory may have more columns, but never more than a single line.
List mode
xz --robot --list uses tab-separated output. The first column of every line has a string that indicates the type of the informa-
tion found on that line:
name This is always the first line when starting to list a file. The second column on the line is the filename.
file This line contains overall information about the .xz file. This line is always printed after the name line.
stream This line type is used only when --verbose was specified. There are as many stream lines as there are streams in the .xz
file.
block This line type is used only when --verbose was specified. There are as many block lines as there are blocks in the .xz
file. The block lines are shown after all the stream lines; different line types are not interleaved.
summary
This line type is used only when --verbose was specified twice. This line is printed after all block lines. Like the
file line, the summary line contains overall information about the .xz file.
totals This line is always the very last line of the list output. It shows the total counts and sizes.
The columns of the file lines:
2. Number of streams in the file
3. Total number of blocks in the stream(s)
4. Compressed size of the file
5. Uncompressed size of the file
6. Compression ratio, for example 0.123. If ratio is over 9.999, three dashes (---) are displayed instead of the ratio.
7. Comma-separated list of integrity check names. The following strings are used for the known check types: None, CRC32,
CRC64, and SHA-256. For unknown check types, Unknown-N is used, where N is the Check ID as a decimal number (one or
two digits).
8. Total size of stream padding in the file
The columns of the stream lines:
2. Stream number (the first stream is 1)
3. Number of blocks in the stream
4. Compressed start offset
5. Uncompressed start offset
6. Compressed size (does not include stream padding)
7. Uncompressed size
8. Compression ratio
9. Name of the integrity check
10. Size of stream padding
The columns of the block lines:
2. Number of the stream containing this block
3. Block number relative to the beginning of the stream (the first block is 1)
4. Block number relative to the beginning of the file
5. Compressed start offset relative to the beginning of the file
6. Uncompressed start offset relative to the beginning of the file
7. Total compressed size of the block (includes headers)
8. Uncompressed size
9. Compression ratio
10. Name of the integrity check
If --verbose was specified twice, additional columns are included on the block lines. These are not displayed with a single
-verbose, because getting this information requires many seeks and can thus be slow:
11. Value of the integrity check in hexadecimal
12. Block header size
13. Block flags: c indicates that compressed size is present, and u indicates that uncompressed size is present. If the
flag is not set, a dash (-) is shown instead to keep the string length fixed. New flags may be added to the end of
the string in the future.
14. Size of the actual compressed data in the block (this excludes the block header, block padding, and check fields)
15. Amount of memory (in bytes) required to decompress this block with this xz version
16. Filter chain. Note that most of the options used at compression time cannot be known, because only the options that
are needed for decompression are stored in the .xz headers.
The columns of the summary lines:
2. Amount of memory (in bytes) required to decompress this file with this xz version
3. yes or no indicating if all block headers have both compressed size and uncompressed size stored in them
Since xz 5.1.2alpha:
4. Minimum xz version required to decompress the file
The columns of the totals line:
2. Number of streams
3. Number of blocks
4. Compressed size
5. Uncompressed size
6. Average compression ratio
7. Comma-separated list of integrity check names that were present in the files
8. Stream padding size
9. Number of files. This is here to keep the order of the earlier columns the same as on file lines.
If --verbose was specified twice, additional columns are included on the totals line:
10. Maximum amount of memory (in bytes) required to decompress the files with this xz version
11. yes or no indicating if all block headers have both compressed size and uncompressed size stored in them
Since xz 5.1.2alpha:
12. Minimum xz version required to decompress the file
Future versions may add new line types and new columns can be added to the existing line types, but the existing columns won't be
changed.
EXIT STATUS
0 All is good.
1 An error occurred.
2 Something worth a warning occurred, but no actual errors occurred.
Notices (not warnings or errors) printed on standard error don't affect the exit status.
ENVIRONMENT
xz parses space-separated lists of options from the environment variables XZ_DEFAULTS and XZ_OPT, in this order, before parsing
the options from the command line. Note that only options are parsed from the environment variables; all non-options are
silently ignored. Parsing is done with getopt_long(3) which is used also for the command line arguments.
XZ_DEFAULTS
User-specific or system-wide default options. Typically this is set in a shell initialization script to enable xz's mem-
ory usage limiter by default. Excluding shell initialization scripts and similar special cases, scripts must never set or
unset XZ_DEFAULTS.
XZ_OPT This is for passing options to xz when it is not possible to set the options directly on the xz command line. This is the
case e.g. when xz is run by a script or tool, e.g. GNU tar(1):
XZ_OPT=-2v tar caf foo.tar.xz foo
Scripts may use XZ_OPT e.g. to set script-specific default compression options. It is still recommended to allow users to
override XZ_OPT if that is reasonable, e.g. in sh(1) scripts one may use something like this:
XZ_OPT=${XZ_OPT-"-7e"}
export XZ_OPT
LZMA UTILS COMPATIBILITY
The command line syntax of xz is practically a superset of lzma, unlzma, and lzcat as found from LZMA Utils 4.32.x. In most
cases, it is possible to replace LZMA Utils with XZ Utils without breaking existing scripts. There are some incompatibilities
though, which may sometimes cause problems.
Compression preset levels
The numbering of the compression level presets is not identical in xz and LZMA Utils. The most important difference is how dic-
tionary sizes are mapped to different presets. Dictionary size is roughly equal to the decompressor memory usage.
Level xz LZMA Utils
-0 256 KiB N/A
-1 1 MiB 64 KiB
-2 2 MiB 1 MiB
-3 4 MiB 512 KiB
-4 4 MiB 1 MiB
-5 8 MiB 2 MiB
-6 8 MiB 4 MiB
-7 16 MiB 8 MiB
-8 32 MiB 16 MiB
-9 64 MiB 32 MiB
The dictionary size differences affect the compressor memory usage too, but there are some other differences between LZMA Utils
and XZ Utils, which make the difference even bigger:
Level xz LZMA Utils 4.32.x
-0 3 MiB N/A
-1 9 MiB 2 MiB
-2 17 MiB 12 MiB
-3 32 MiB 12 MiB
-4 48 MiB 16 MiB
-5 94 MiB 26 MiB
-6 94 MiB 45 MiB
-7 186 MiB 83 MiB
-8 370 MiB 159 MiB
-9 674 MiB 311 MiB
The default preset level in LZMA Utils is -7 while in XZ Utils it is -6, so both use an 8 MiB dictionary by default.
Streamed vs. non-streamed .lzma files
The uncompressed size of the file can be stored in the .lzma header. LZMA Utils does that when compressing regular files. The
alternative is to mark that uncompressed size is unknown and use end-of-payload marker to indicate where the decompressor should
stop. LZMA Utils uses this method when uncompressed size isn't known, which is the case for example in pipes.
xz supports decompressing .lzma files with or without end-of-payload marker, but all .lzma files created by xz will use end-of-
payload marker and have uncompressed size marked as unknown in the .lzma header. This may be a problem in some uncommon situa-
tions. For example, a .lzma decompressor in an embedded device might work only with files that have known uncompressed size. If
you hit this problem, you need to use LZMA Utils or LZMA SDK to create .lzma files with known uncompressed size.
Unsupported .lzma files
The .lzma format allows lc values up to 8, and lp values up to 4. LZMA Utils can decompress files with any lc and lp, but always
creates files with lc=3 and lp=0. Creating files with other lc and lp is possible with xz and with LZMA SDK.
The implementation of the LZMA1 filter in liblzma requires that the sum of lc and lp must not exceed 4. Thus, .lzma files, which
exceed this limitation, cannot be decompressed with xz.
LZMA Utils creates only .lzma files which have a dictionary size of 2^n (a power of 2) but accepts files with any dictionary
size. liblzma accepts only .lzma files which have a dictionary size of 2^n or 2^n + 2^(n-1). This is to decrease false posi-
tives when detecting .lzma files.
These limitations shouldn't be a problem in practice, since practically all .lzma files have been compressed with settings that
liblzma will accept.
Trailing garbage
When decompressing, LZMA Utils silently ignore everything after the first .lzma stream. In most situations, this is a bug. This
also means that LZMA Utils don't support decompressing concatenated .lzma files.
If there is data left after the first .lzma stream, xz considers the file to be corrupt unless --single-stream was used. This
may break obscure scripts which have assumed that trailing garbage is ignored.
NOTES
Compressed output may vary
The exact compressed output produced from the same uncompressed input file may vary between XZ Utils versions even if compression
options are identical. This is because the encoder can be improved (faster or better compression) without affecting the file
format. The output can vary even between different builds of the same XZ Utils version, if different build options are used.
The above means that once --rsyncable has been implemented, the resulting files won't necessarily be rsyncable unless both old
and new files have been compressed with the same xz version. This problem can be fixed if a part of the encoder implementation
is frozen to keep rsyncable output stable across xz versions.
Embedded .xz decompressors
Embedded .xz decompressor implementations like XZ Embedded don't necessarily support files created with integrity check types
other than none and crc32. Since the default is --check=crc64, you must use --check=none or --check=crc32 when creating files
for embedded systems.
Outside embedded systems, all .xz format decompressors support all the check types, or at least are able to decompress the file
without verifying the integrity check if the particular check is not supported.
XZ Embedded supports BCJ filters, but only with the default start offset.
EXAMPLES
Basics
Compress the file foo into foo.xz using the default compression level (-6), and remove foo if compression is successful:
xz foo
Decompress bar.xz into bar and don't remove bar.xz even if decompression is successful:
xz -dk bar.xz
Create baz.tar.xz with the preset -4e (-4 --extreme), which is slower than e.g. the default -6, but needs less memory for com-
pression and decompression (48 MiB and 5 MiB, respectively):
tar cf - baz | xz -4e > baz.tar.xz
A mix of compressed and uncompressed files can be decompressed to standard output with a single command:
xz -dcf a.txt b.txt.xz c.txt d.txt.lzma > abcd.txt
Parallel compression of many files
On GNU and *BSD, find(1) and xargs(1) can be used to parallelize compression of many files:
find . -type f \! -name '*.xz' -print0 \
| xargs -0r -P4 -n16 xz -T1
The -P option to xargs(1) sets the number of parallel xz processes. The best value for the -n option depends on how many files
there are to be compressed. If there are only a couple of files, the value should probably be 1; with tens of thousands of
files, 100 or even more may be appropriate to reduce the number of xz processes that xargs(1) will eventually create.
The option -T1 for xz is there to force it to single-threaded mode, because xargs(1) is used to control the amount of paral-
lelization.
Robot mode
Calculate how many bytes have been saved in total after compressing multiple files:
xz --robot --list *.xz | awk '/^totals/{print $5-$4}'
A script may want to know that it is using new enough xz. The following sh(1) script checks that the version number of the xz
tool is at least 5.0.0. This method is compatible with old beta versions, which didn't support the --robot option:
if ! eval "$(xz --robot --version 2> /dev/null)" ||
[ "$XZ_VERSION" -lt 50000002 ]; then
echo "Your xz is too old."
fi
unset XZ_VERSION LIBLZMA_VERSION
Set a memory usage limit for decompression using XZ_OPT, but if a limit has already been set, don't increase it:
NEWLIM=$((123 << 20)) # 123 MiB
OLDLIM=$(xz --robot --info-memory | cut -f3)
if [ $OLDLIM -eq 0 -o $OLDLIM -gt $NEWLIM ]; then
XZ_OPT="$XZ_OPT --memlimit-decompress=$NEWLIM"
export XZ_OPT
fi
Custom compressor filter chains
The simplest use for custom filter chains is customizing a LZMA2 preset. This can be useful, because the presets cover only a
subset of the potentially useful combinations of compression settings.
The CompCPU columns of the tables from the descriptions of the options -0 ... -9 and --extreme are useful when customizing LZMA2
presets. Here are the relevant parts collected from those two tables:
Preset CompCPU
-0 0
-1 1
-2 2
-3 3
-4 4
-5 5
-6 6
-5e 7
-6e 8
If you know that a file requires somewhat big dictionary (e.g. 32 MiB) to compress well, but you want to compress it quicker than
xz -8 would do, a preset with a low CompCPU value (e.g. 1) can be modified to use a bigger dictionary:
xz --lzma2=preset=1,dict=32MiB foo.tar
With certain files, the above command may be faster than xz -6 while compressing significantly better. However, it must be em-
phasized that only some files benefit from a big dictionary while keeping the CompCPU value low. The most obvious situation,
where a big dictionary can help a lot, is an archive containing very similar files of at least a few megabytes each. The dictio-
nary size has to be significantly bigger than any individual file to allow LZMA2 to take full advantage of the similarities be-
tween consecutive files.
If very high compressor and decompressor memory usage is fine, and the file being compressed is at least several hundred
megabytes, it may be useful to use an even bigger dictionary than the 64 MiB that xz -9 would use:
xz -vv --lzma2=dict=192MiB big_foo.tar
Using -vv (--verbose --verbose) like in the above example can be useful to see the memory requirements of the compressor and de-
compressor. Remember that using a dictionary bigger than the size of the uncompressed file is waste of memory, so the above com-
mand isn't useful for small files.
Sometimes the compression time doesn't matter, but the decompressor memory usage has to be kept low e.g. to make it possible to
decompress the file on an embedded system. The following command uses -6e (-6 --extreme) as a base and sets the dictionary to
only 64 KiB. The resulting file can be decompressed with XZ Embedded (that's why there is --check=crc32) using about 100 KiB of
memory.
xz --check=crc32 --lzma2=preset=6e,dict=64KiB foo
If you want to squeeze out as many bytes as possible, adjusting the number of literal context bits (lc) and number of position
bits (pb) can sometimes help. Adjusting the number of literal position bits (lp) might help too, but usually lc and pb are more
important. E.g. a source code archive contains mostly US-ASCII text, so something like the following might give slightly (like
0.1 %) smaller file than xz -6e (try also without lc=4):
xz --lzma2=preset=6e,pb=0,lc=4 source_code.tar
Using another filter together with LZMA2 can improve compression with certain file types. E.g. to compress a x86-32 or x86-64
shared library using the x86 BCJ filter:
xz --x86 --lzma2 libfoo.so
Note that the order of the filter options is significant. If --x86 is specified after --lzma2, xz will give an error, because
there cannot be any filter after LZMA2, and also because the x86 BCJ filter cannot be used as the last filter in the chain.
The Delta filter together with LZMA2 can give good results with bitmap images. It should usually beat PNG, which has a few more
advanced filters than simple delta but uses Deflate for the actual compression.
The image has to be saved in uncompressed format, e.g. as uncompressed TIFF. The distance parameter of the Delta filter is set
to match the number of bytes per pixel in the image. E.g. 24-bit RGB bitmap needs dist=3, and it is also good to pass pb=0 to
LZMA2 to accommodate the three-byte alignment:
xz --delta=dist=3 --lzma2=pb=0 foo.tiff
If multiple images have been put into a single archive (e.g. .tar), the Delta filter will work on that too as long as all images
have the same number of bytes per pixel.
example:
for an 8GB rpi SD image it to twice as long to compresss as gzip BUT only 2/3 the size !!
SEE ALSO
xzdec(1), xzdiff(1), xzgrep(1), xzless(1), xzmore(1), gzip(1), bzip2(1), 7z(1)
XZ Utils:
LZMA SDK:
Tukaani 2017-04-19