Parallel BZIP2 (PBZIP2)
Data Compression Software
by Jeff Gilchrist
NOTE: If you are looking for a parallel BZIP2 that works on cluster machines, you should check out MPIBZIP2 which was designed for a distributed-memory message-passing architecture. Screen Shot
License/Disclaimer
This software is distributed under a BSD-style license. For details, see the file COPYING. Use at your own risk. I take no responsibility for anything that happens to your data or equipment. Always test (bzip2 -tv) a compressed file containing important data before deleting the original to verify the compression was successful.
If you find this software useful or you are using it in a government/business/commercial environment, please consider making a donation to help support future improvements:
Download
Recent History
Contributions
- Bryan Stillwell <bryan [at] bokeoa {dot} com> - code cleanup, RPM spec, and prep work for inclusion in Fedora Extras - Dru Lemley [http://lemley.net/smp.html] - help with large file support - Kir Kolyshkin <kir [at] sacred {dot} ru> - autodetection for # of CPUs - Joergen Ramskov <joergen [at] ramskov {dot} org> - initial version of man page - Peter Cordes <peter [at] cordes {dot} ca> - code cleanup - Kurt Fitzner <kfitzner [at] excelcia {dot} org> - port to Windows compilers and decompression throttling - Oliver Falk <oliver [at] linux-kernel {dot} at> - RPM spec update - Jindrich Novy <jnovy [at] redhat {dot} com> - code cleanup and bug fixes - Benjamin Reed <ranger [at] befunk {dot} com> - autodetection for # of CPUs in OSX and maintains OSX packages - Chris Dearman <chris [at] mips {dot} com> - fixed pthreads race condition that led to pthread resources issues when processing large numbers of files and random segfaults - Richard Russon <ntfs [at] flatcap {dot} org> - help fix decompression bug - Paul Pluzhnikov <paul [at] parasoft {dot} com> - fixed minor memory leak - Anibal Monsalve Salazar <anibal [at] debian {dot} org> - creates and maintains Debian packages - Steve Christensen - creates and maintains Solaris packages (sunfreeware.com) - Alessio Cervellin - creates and maintains Solaris packages (blastwave.org) - Andre Przywara - creates and maintains Slackware packages (linuxpackages.net) - Ying-Chieh Liao - created the FreeBSD port - Andrew Pantyukhin <sat [at] FreeBSD {dot} org> - maintains the FreeBSD port and willing to resolve any FreeBSD-related problems - Roland Illig - creates and maintains the NetBSD packages - Matt Turner <mattst88 [at] gmail {dot} com> - code cleanup - Alvaro Reguly <alvaro [at] reguly {dot} com> - RPM spec update to support SUSE Linux - Ivan Voras <ivoras [at] freebsd {dot} org> - support for stdin and pipes during compression and CPU detect changes - John Dalton <john [at] johndalton {dot} info> - code cleanup and bug fix for stdin support - Rene Georgi <rene.georgi [at] online {dot} de> - code and Makefile cleanup, support for direct decompress and bzcat - Rene Rheaume & Jeroen Roovers <jer [at] xs4all {dot} nl> - patch to support uclibc's lack of a getloadavg function - Reinhard Schiedermeier <rs [at] cs {dot} hm {dot} edu> - support for tar --use-compress-prog=pbzip2 - Elbert Pol - creates and maintains OS/2 packages- Nico Vrouwe <nico [at] gojelly {dot} com> - support for CPU detection on Windows- Eduardo Terol <EduardoTerol [at] gmx {dot} net> - creates and maintains Windows 32bit package
Special Thanks for suggestions and testing to: Phillippe Welsh, Cassens Transport Co., James Terhune, Dru Lemley, Bryan Stillwell, George Chalissery, Kir Kolyshkin, Madhu Kangara, Mike Furr, Joergen Ramskov, Kurt Fitzner, Peter Cordes, Oliver Falk, Jindrich Novy, Benjamin Reed, Chris Dearman, Richard Russon, Anibal Monsalve Salazar, Jim Leonard, Paul Pluzhniko, Robert Archard, Coran Fisher, Ken Takusagawa, David Pyke, Matt Turner, Damien Ancelin, Alvaro Reguly, Ivan Voras, John Dalton, Sami Liedes, Rene Georgi, Rene Rheaume, Jeroen Roovers, Reinhard Schiedermeier, Kari Pahula, Elbert Pol, Nico Vrouwe, Eduardo Terol.
ToDo
- Add multi-threaded support for decompression from stdin & pipes Benchmark Results
The following benchmark was performed with various systems using a 900k block size. The pbzip2 software was benchmarked with the Intanium2, Opteron, and Xeon processors using a Linux 2.6 64bit kernel while the Core2 used Windows Vista 64bit (cygwin).
For more benchmark information click here.
Usage
Run pbzip2 for the help listing. ===================================================================
Usage: pbzip2 [-1 .. -9] [-b#cdfhklp#qrtvVz] <filename> <filename2> <filenameN> -b#: where # is the file block size in 100k (default 9 = 900k) -c : output to standard out (stdout) -d : decompress file -f : force, overwrite existing output file -h : print this help message -k : keep input file, don't delete -l : load average determines max number processors to use -p#: where # is the number of processors (default: autodetect [2]) -q : quiet mode (default) -r : read entire input file into RAM and split between processors -t : test compressed file integrity -v : verbose mode -V : display version info for pbzip2 then exit -z : compress file (default) -1 .. -9 : set BWT block size to 100k .. 900k (default 900k)Example: pbzip2 -b15qk myfile.tarExample: pbzip2 -p4 -r -5 myfile.tar second*.txtExample: tar -c directory_to_compress/ | pbzip2 -c > myfile.tar.bz2Example: pbzip2 -d myfile.tar.bz2
=================================================================== The pbzip2 program is a parallel version of bzip2 for use on shared memory machines. It provides near-linear speedup when used on true multi-processor machines and 5-10% speedup on Hyperthreaded machines. The output is fully compatible with the regular bzip2 data so any files created with pbzip2 can be uncompressed by bzip2 and vice-versa.
The default settings for pbzip2 will work well in most cases. The only switch you will likely need to use is -d to decompress files and -p to set the # of processors for pbzip2 to use if autodetect is not supported on your system, or you want to use a specific # of CPUs. Example 1: pbzip2 -v myfile.tar This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k.
The program would report something like: ===================================================================
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca][Jan. 08, 2009] (uses libbzip2 by Julian Seward) # CPUs: 2 BWT Block Size: 900kFile Block Size: 900k------------------------------------------- File #: 1 of 1 Input Name: myfile.tarOutput Name: myfile.tar.bz2 Input Size: 7428687 bytesCompressing data...Output Size: 3236549 bytes------------------------------------------- Wall Clock: 2.809000 seconds
=================================================================== Example 2: pbzip2 -b15vk myfile.tar This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with a file block size of 1500k and a BWT block size of 900k. The file "myfile.tar" will not be deleted after compression is finished.
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca][Jan. 08, 2009] (uses libbzip2 by Julian Seward) # CPUs: 2 BWT Block Size: 900kFile Block Size: 1500k------------------------------------------- File #: 1 of 1 Input Name: myfile.tar Output Name: myfile.tar.bz2 Input Size: 7428687 bytesCompressing data... Output Size: 3236394 bytes------------------------------------------- Wall Clock: 3.059000 seconds
=================================================================== Example 3: pbzip2 -p4 -r -5 -v myfile.tar second*.txt This example will compress the file "myfile.tar" into the compressed file "myfile.tar.bz2". It will use 4 processors with a BWT block size of 500k. The file block size will be the size of "myfile.tar" divided by 4 (# of processors) so that the data will be split evenly among each processor. This requires you have enough RAM for pbzip2 to read the entire file into memory for compression. Pbzip2 will then use the same options to compress all other files that match the wildcard "second*.txt" in that directory.
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca][Jan. 08, 2009] (uses libbzip2 by Julian Seward) # CPUs: 4 BWT Block Size: 500kFile Block Size: 1857k------------------------------------------- File #: 1 of 3 Input Name: myfile.tar Output Name: myfile.tar.bz2 Input Size: 7428687 bytesCompressing data... Output Size: 3237105 bytes------------------------------------------- File #: 2 of 3 Input Name: secondfile.txt Output Name: secondfile.txt.bz2 Input Size: 5897 bytesCompressing data... Output Size: 3192 bytes ------------------------------------------- File #: 3 of 3 Input Name: secondbreakfast.txt Output Name: secondbreakfast.txt.bz2 Input Size: 83531 bytesCompressing data... Output Size: 11832 bytes ------------------------------------------- Wall Clock: 5.127381 seconds
===================================================================
Example 4: tar cf myfile.tar.bz2 --use-compress-prog=pbzip2 dir_to_compress/ Example 4: tar -c directory_to_compress/ | pbzip2 -vc > myfile.tar.bz2 This example will compress the data being given to pbzip2 via pipe from TAR into the compressed file "myfile.tar.bz2". It will use the autodetected # of processors (or 2 processors if autodetect not supported) with the default file block size of 900k and default BWT block size of 900k. TAR is collecting all of the files from the "directory_to_compress/" directory and passing the data to pbzip2 as it works.
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca][Jan. 08, 2009] (uses libbzip2 by Julian Seward) # CPUs: 2 BWT Block Size: 900kFile Block Size: 900k------------------------------------------- File #: 1 of 1 Input Name: <stdin>Output Name: <stdout>Compressing data...------------------------------------------- Wall Clock: 0.176441 seconds
=================================================================== Example 5: pbzip2 -dv myfile.tar.bz2 This example will decompress the file "myfile.tar.bz2" into the decompressed file "myfile.tar". It will use the autodetected # of processors (or 2 processors if autodetect not supported). The switches -b, -r, and -1..-9 are not valid for decompression.
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca][Jan. 08, 2009] (uses libbzip2 by Julian Seward) # CPUs: 2------------------------------------------- File #: 1 of 1 Input Name: myfile.tar.bz2 Output Name: myfile.tar BWT Block Size: 900k Input Size: 3236549 bytesDecompressing data... Output Size: 7428687 bytes------------------------------------------- Wall Clock: 1.154000 seconds
=================================================================== Bugs/Contact
If you would like to report any bugs or contact me related to the software you can reach me via e-mail at: