Comments on: ChopZip: a parallel implementation of arbitrary compression algorithms http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/ Because FLOSS is handy, isn't it? Mon, 21 May 2012 06:39:20 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Nagilum http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-116107 Nagilum Sat, 14 Jan 2012 12:37:50 +0000 http://handyfloss.net/?p=913#comment-116107 For gzip there is pigz.. To build pxz you need to install liblzma-dev. For gzip there is pigz..
To build pxz you need to install liblzma-dev.

]]> By: isilanes http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-95593 isilanes Mon, 22 Aug 2011 12:44:48 +0000 http://handyfloss.net/?p=913#comment-95593 Responding to my own comment above, I have found the reason for the dramatic speedup of parallel. When compressing data strings, compressors look for redundancy of data (if no redundancy is present, no compression is possible, because by definition our data can not be conveyed by a smaller set of data. If it could, it would mean some of it is redundant). Modern compressors do not search for such redundancies in the whole input data stream, because that would skyrocket the usage of memory and CPU. Instead of that, they compress the input data in chunks: they get the first MB of data and compress it, then the second MB, etc (or whatever chunk size they use). The catch here is that parallel splits the input stream into 1 MB chunks by default, and each chunk is send to a single process (xz, in our example). When making a serial compression, apparently longer chunks are used (xz knows that the bigger the chunks, the more CPU and memory will be used, but also the better the compression ratio, so it uses as much as it can), so it's slower (but compresses more) than with parallel. We can force parallel to split the input data into bigger chunks before sending each chunk to a xz process, as follows: $ cat inputfile | parallel --pipe -k --blocksize 2000k xz -3 > inputfile.xz Increasing the blocksize to around 3200 kB, we get a 8 second compression, which is a quarter of the serial compression. Bigger chunks result in slower compression, and smaller ones in a faster one (up to a limit). Also, increasing the block size to 3200 kB does reduce the size of the compressed file, actually to the level obtained by ChopZip or serial xz. Responding to my own comment above, I have found the reason for the dramatic speedup of parallel. When compressing data strings, compressors look for redundancy of data (if no redundancy is present, no compression is possible, because by definition our data can not be conveyed by a smaller set of data. If it could, it would mean some of it is redundant). Modern compressors do not search for such redundancies in the whole input data stream, because that would skyrocket the usage of memory and CPU. Instead of that, they compress the input data in chunks: they get the first MB of data and compress it, then the second MB, etc (or whatever chunk size they use).

The catch here is that parallel splits the input stream into 1 MB chunks by default, and each chunk is send to a single process (xz, in our example). When making a serial compression, apparently longer chunks are used (xz knows that the bigger the chunks, the more CPU and memory will be used, but also the better the compression ratio, so it uses as much as it can), so it’s slower (but compresses more) than with parallel.

We can force parallel to split the input data into bigger chunks before sending each chunk to a xz process, as follows:

$ cat inputfile | parallel –pipe -k –blocksize 2000k xz -3 > inputfile.xz

Increasing the blocksize to around 3200 kB, we get a 8 second compression, which is a quarter of the serial compression. Bigger chunks result in slower compression, and smaller ones in a faster one (up to a limit). Also, increasing the block size to 3200 kB does reduce the size of the compressed file, actually to the level obtained by ChopZip or serial xz.

]]> By: isilanes http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-95441 isilanes Fri, 19 Aug 2011 09:03:35 +0000 http://handyfloss.net/?p=913#comment-95441 Thanks Ole, I seems that it could certainly do. However I am getting somewhat disturbing results with it. With either xz, xz + parallel, or ChopZip, I compress a given file, then decompress it back and check that the md5sum hasn't changed, so the compression and decompression where correct. Now the file sizes are: Uncompressed: 94 MB (97922080 bytes) xz -3: 14399948 bytes xz -3 + parallel -j 4: 14767356 bytes chopzip.py -n 4 -m xz -l 3: 14413616 bytes So far, so good. Parallel compression is slightly less efficient than serial, which was expected. I still don't know why chopzip beats xz + parallel in size, if they both split the input into 4 equal pieces, but oh well... However, the surprise comes when benchmarking the speed. The compression times are: xz: 32.8 s chopzip.py: 14.6 s xz + parallel: 3.77 s I am disapointed at how chopzip barely gets a 2x speedup, when 4x is expected. However, I am much much more surprised at how xz+parallel got almost a 9x speedup!! I just don't get it... I also tried gzip, instead of xz, and with it gzip+parallel was not much faster than serial gzip (1.33 vs 1.83 s). I'm a bit puzzled. Thanks Ole,

I seems that it could certainly do.

However I am getting somewhat disturbing results with it. With either xz, xz + parallel, or ChopZip, I compress a given file, then decompress it back and check that the md5sum hasn’t changed, so the compression and decompression where correct. Now the file sizes are:

Uncompressed: 94 MB (97922080 bytes)
xz -3: 14399948 bytes
xz -3 + parallel -j 4: 14767356 bytes
chopzip.py -n 4 -m xz -l 3: 14413616 bytes

So far, so good. Parallel compression is slightly less efficient than serial, which was expected. I still don’t know why chopzip beats xz + parallel in size, if they both split the input into 4 equal pieces, but oh well…

However, the surprise comes when benchmarking the speed. The compression times are:

xz: 32.8 s
chopzip.py: 14.6 s
xz + parallel: 3.77 s

I am disapointed at how chopzip barely gets a 2x speedup, when 4x is expected. However, I am much much more surprised at how xz+parallel got almost a 9x speedup!! I just don’t get it…

I also tried gzip, instead of xz, and with it gzip+parallel was not much faster than serial gzip (1.33 vs 1.83 s). I’m a bit puzzled.

]]> By: Ole Tange http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-95405 Ole Tange Thu, 18 Aug 2011 12:25:43 +0000 http://handyfloss.net/?p=913#comment-95405 Could ChopZip be replaced by GNU Parallel: ZIPPRG=lzip cat bigfile | parallel --pipe -k $ZIPPRG > bigfile.lz Could ChopZip be replaced by GNU Parallel:

ZIPPRG=lzip
cat bigfile | parallel –pipe -k $ZIPPRG > bigfile.lz

]]> By: isilanes http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-48960 isilanes Tue, 05 Jan 2010 11:10:01 +0000 http://handyfloss.net/?p=913#comment-48960 Thanks, Iñigo! I also experimented with [[pexec]]. I even made a Python function to mimic pexec (and I guess ParallelPython), in the sense that it accepts a list of commands, then runs N of them simultaneously, making sure that there are always N processes running (for N cores). When one finishes, a new one is added to the list of running processes, until the list of commands to run is exhausted. In the case of ChopZip, I ended up going back to my original formulation, to avoid unnecessary code complexity (for little or no efficiency gain). pexec-like functionality allows (in my case) for generating more file pieces than cores are available, and thus a better load balance (compensate for chunks that get compressed faster, and their core would be idle). However, as I said, I saw no actual performance gain. Thanks, Iñigo! I also experimented with pexec. I even made a Python function to mimic pexec (and I guess ParallelPython), in the sense that it accepts a list of commands, then runs N of them simultaneously, making sure that there are always N processes running (for N cores). When one finishes, a new one is added to the list of running processes, until the list of commands to run is exhausted. In the case of ChopZip, I ended up going back to my original formulation, to avoid unnecessary code complexity (for little or no efficiency gain). pexec-like functionality allows (in my case) for generating more file pieces than cores are available, and thus a better load balance (compensate for chunks that get compressed faster, and their core would be idle). However, as I said, I saw no actual performance gain.

]]> By: Inigo http://handyfloss.net/2009.12/chopzip-a-parallel-implementation-of-arbitrary-compression-algorithms/comment-page-1/#comment-47606 Inigo Thu, 24 Dec 2009 02:36:04 +0000 http://handyfloss.net/?p=913#comment-47606 For trivial paralelization using python, with some examples that may ring to you, take a look at Parallel Python http://www.parallelpython.com/ Iñigo For trivial paralelization using python, with some examples that may ring to you, take a look at Parallel Python http://www.parallelpython.com/

Iñigo

]]>