How to Parallelize Compression in Ubuntu Linux

Compressing large files on Ubuntu often bottlenecks at a single CPU core when using standard tools. This guide demonstrates how to enable multi-threaded compression to significantly reduce processing time. We will cover installing parallel utilities like pigz and configuring existing tools like xz to utilize all available hardware resources efficiently.

Standard compression tools like gzip are single-threaded by default, meaning they only use one CPU core. To speed this up, you can use parallel implementations that split the workload across multiple cores.

Install Parallel Tools

Open your terminal and update your package list. Then install pigz and pbzip2 using the following command: sudo apt update && sudo apt install pigz pbzip2

Compress with Pigz

Pigz is a parallel implementation of gzip that produces identical output. To compress a file using all available cores, run: pigz file.tar To specify the number of threads manually, use the -p flag: pigz -p 4 file.tar

Compress with XZ

The xz tool supports parallel compression using the -T option. To use all available cores, execute: xz -T 0 file.tar Setting the thread count to 0 allows xz to automatically detect and use all online processors.

Compress with PBzip2

For bzip2 archives, pbzip2 offers multi-threading support. The usage is similar to standard bzip2: pbzip2 -p4 file.tar This command compresses the file using four specific threads.

Create Aliases for Convenience

To make parallel compression the default behavior, add aliases to your .bashrc file. Open the configuration file: nano ~/.bashrc Add the following lines to the end of the file: alias gzip='pigz' alias bzip2='pbzip2' Save the file and reload your shell configuration with source ~/.bashrc. Now, standard compression commands will automatically utilize multiple cores.