Pigz – Compress And Decompress Files In Parallel In Linux

Pigz, short for parallel implementation of gzip, is a free, open source multi-threaded compression utility to compress and uncompress files in Linux. Pigz, pronounced as pig-zee, uses the zlib and pthread libraries and takes full advantage of multiple processors and multiple cores when compressing data. Since pigz compresses using threads to make use of multiple processors and cores, It can be able to archive larger files much faster than with gzip. To put this simply, pigz does what gzip does, but it allocates the work to multiple processors and cores when compressing and speed up the compression/decompression process significantly. In this guide, let us learn to compress and decompress files in parallel using Pigz in Linux.

Install Pigz in Linux

Pigz is available in the official repositories of popular Linux operating systems.

On Arch Linux and its variants like Manjaro Linux, enable [Community] repository and install Pigz using command:

$ sudo pacman -S pigz

On CentOS 8/7/6:

$ sudo yum install pigz

On Debian:

$ sudo apt-get install pigz

On Fedora:

$ sudo dnf install pigz

On Ubuntu, Linux Mint:

$ sudo add-apt-repository universe

$ sudo apt install pigz

On openSUSE:

$ sudo zypper install pigz

On other Linux distributions, compile and install from source as shown below.

First, make sure you have installed the following prerequisites:

• make
• gcc
• zlib-devel

For example, on Debian-based systems you can install the above packages using command:

$ sudo apt install make gcc zlib-devel

On RPM-based systems:

$ sudo yum install make gcc zlib-devel

Then download the latest Pigz version from the official website (Link is given at the end):

$ wget https://zlib.net/pigz/pigz-2.4.tar.gz

Extract the pigz tar file:

$ tar -xzvf pigz-2.4.tar.gz

This command will extract the contents in a folder named pigz-2.4 in the current working directory.

CD into "pigz-2.4" directory:

$ cd pigz-2.4

Compile and install it using the following command:

$ make

Copy the pigz binary to your $PATH, for example /usr/local/bin:

$ sudo cp pigz /usr/local/bin/

Log out and log back in to start using pigz.

Compress And Decompress Files Using Pigz

Pigz divides the given input into 128 KB chunks and compresses each one of them in parallel. The compressed data format generated is in the gzip, zlib, or single-entry zip format using the deflate compression method. By default, it compresses the file gzip (.gz) format.

Let me show you some practical examples. For the purpose of this guide, I will be using archlinux.iso file (size 677.4 MB) for testing.

Compress files

To compress a file with default options, simply run:

pigz <path-to-file>

Example:

$ pigz archlinux.iso

The above command will compress the given file i.e. archlinux.iso and save it as archlinux.iso.gz in the current working directory.

Please note that the above command will delete the original file i.e. archlinux.iso after compressing it. If you don't want to delete the original file, use -k (keep) to instruct Pigz not to delete the original file after processing it.

$ pigz -k archlinux.iso

List the contents of an archive file

To list the contents of the above archived file, without extracting it, use -l flag.

$ pigz -l archlinux.iso.gz 
compressed   original reduced  name
 654901763  677380096    3.3%  archlinux.iso

As you can see in the above output, the input file has been reduced by 3.3% from its original size.

Specify compression methods

Pigz supports the following compression methods:

• Slowest and best compression (-9)
• Fastest and less compression (-1)
• No compression (-0)
• Default compression (-6)

For example, to compress a file using the best compression method (slow), run:

$ pigz -9 archlinux.iso

Compress a file with less compression (fast):

$ pigz -1 archlinux.iso

No compression:

$ pigz -0 archlinux.iso

Change compression format

By default, it saves the output file in gzip format. You can, however, change it to different format.

The following command compresses the given file to zlib (.zz) instead of gzip format:

$ pigz -k -z archlinux.iso

Here, we used the -k (lower case) to keep the original file after compressing it.

Similarly, to compress the file in zip format, use -K (upper case) flag:

$ pigz -k -K archlinux.iso

Compress Directories

Pigz doesn't have any option to directly compress folders. However, we can achieve this by combining tar command with pigz like below.

$ tar cf - Pictures/ | pigz > pictures.tar.gz

The above command will compress Pictures folder and saves it as pictures.tar.gz file.

As one of our reader Wazz said in the comment section below, we can also use the ‘use-compress-program’ parameter within tar to run pigz instead of piping it out.

Example:

$ tar --use-compress-program=pigz -cf yourfile.tar.gz folder1 folder2

Limit number of processors while compressing

Like I already said, Pigz exploits multiple processors and cores to the hilt when compressing files. You can change this behaviour using -p flag.

For example the following command will compress a file using best compression method and 4 processors and kep the original file:

$ pigz -9 -k -p4 archlinux.iso

To find the number of processor cores in Linux, refer the following guide:

• How To Find Number Of CPU Cores From Commandline In Linux

Decompress files

We can decompress the compressed files to their original form using pigz -d or unpigz commands.

$ pigz -d archlinux.iso.gz

Or,

$ unpigz archlinux.iso.gz

For more details, refer the help section and man pages:

$ pigz --help

$  man pigz

Gzip vs Pigz - Performance comparison

I tested the archlinux.iso (size 677.4 MB) file with Pigz And Gzip utilities. Here are the results:

Compression using Gzip:

$ time gzip archlinux.iso

real    0m35.086s
user    0m32.719s
sys 0m1.333s

Compression using Pigz:

$ time pigz archlinux.iso

real 0m12.545s
user 0m46.376s
sys 0m1.520s

Decompression with Gzip:

$ time gzip -d archlinux.iso.gz 

real    0m6.847s
user    0m5.621s
sys 0m0.783s

Decompression with Pigz:

$ time unpigz archlinux.iso.gz 

real    0m2.398s
user    0m2.370s
sys 0m1.931s

As you can see in the above outputs, Pigz can compress and decompress a single file of 677.4 MB two times faster than with gzip.

So, if you have a modern multi-processor, multi-core system, and want to compress larger files as fast as possible, by using all of your available CPU cores, pigz will be a good choice! Give it a try and let us know your thoughts about Pigz compression utility in comment section below.

Related read:

• How To Compress And Decompress Files In Linux
• How To Archive Files And Directories In Linux

Resources:

• Pigz Website
• Pigz GitHub Repository