compression_test/README.md

Compression test

• Compression test
  • Test setup
    • Commands
  • Results
    • Artificial
    • Calgary
    • Canterbury
    • Large

I made some test on multiple of the compression algorithms and tools found on linux.

Test setup

The algorithms have been tested with the canterbury corpus. Downloaded from this repository.

The algorithms tested were the following:

• XZ
• LZMA
• PGZIP (parallel gzip)
• PBZIP2 (parallel bzip2)
• LZ4
• PLZIP (parallel lzip)
• ZSTD
• RAR
• 7ZIP
• ZIP

The tests were run on a system with the following specs

• Ryzen 9 3900X CPU
• 32 GB DDR4 2133 MT/s memory
• nvme gen4 ssd
• Arch linux kernel version 6.12.8
• xfs file system

Notes: Algorithms noyed with (Best compression) were run with the options for best compression otherwise default options are used.

Commands

These were the commands used for each algorithm:

Algorithm	Command
7ZIP	7z a <archive> <dir>
LZ4	tar c -I"lz4" -f <archive> <dir>
LZ4 (Best compression)	tar c -I"lz4 -12" -f <archive> <dir>
LZMA	tar c -I"lzma -T0" -f <archive> <dir>
LZMA (Best compression)	tar c -I"lzma -9 -T0" -f <archive> <dir>
PBZIP2	tar c -Ipbzip2 -f <archive> <dir>
PGZIP	tar c -I"pigz" -f <archive> <dir>
PGZIP (Best compression)	tar c -I"pigz --best" -f <archive> <dir>
PLZIP	tar c -I"plzip" -f <archive> <dir>
PLZIP (Best compression)	tar c -I"plzip -9" -f <archive> <dir>
RAR	rar a <archive> <dir>
XZ	tar c -I"xz -T0" -f <archive> <dir>
XZ (Best compression)	tar c -I"xz -9 -T0" -f <archive> <dir>
ZIP	zip -r <archive> <dir>
ZSTD	tar c -I"zstd -T0" -f <archive> <dir>
ZSTD (Best compression)	tar c -I"zstd -19 -T0" -f <archive> <dir>

Results

All the tables have the next categories:

• Time: The time the compression process took. Lower is better.
• Compession: The compression rate after the process. Higher is better.
• Time normalized compression: The compression normalized to the time obtained ny dividing the compression by the time. Higher is better.

Artificial

Test with "Artificial" sorted by "Time"

Compression Algorithm	Time	Compression	Time normalized compression
LZ4	0.00500178	2.94217	588.224
PGZIP	0.00692916	3.89652	562.336
ZSTD	0.00734138	3.96104	539.549
LZ4 (Best compression)	0.00752068	2.94896	392.114
ZIP	0.00768423	3.87759	504.617
PGZIP (Best compression)	0.00786114	3.89748	495.79
7ZIP	0.0215967	3.87664	179.502
PLZIP	0.0232768	3.88643	166.966
RAR	0.0236309	3.84245	162.603
ZSTD (Best compression)	0.0301487	3.96554	131.532
XZ	0.030818	3.88426	126.039
LZMA	0.0315187	3.88753	123.341
PBZIP2	0.0370195	3.9214	105.928
PLZIP (Best compression)	0.040936	3.88733	94.9612
XZ (Best compression)	0.0605237	3.88426	64.1775
LZMA (Best compression)	0.0609977	3.88753	63.7324

Test with "Artificial" sorted by "Compression"

Compression Algorithm	Time	Compression	Time normalized compression
ZSTD (Best compression)	0.0301487	3.96554	131.532
ZSTD	0.00734138	3.96104	539.549
PBZIP2	0.0370195	3.9214	105.928
PGZIP (Best compression)	0.00786114	3.89748	495.79
PGZIP	0.00692916	3.89652	562.336
LZMA (Best compression)	0.0609977	3.88753	63.7324
LZMA	0.0315187	3.88753	123.341
PLZIP (Best compression)	0.040936	3.88733	94.9612
PLZIP	0.0232768	3.88643	166.966
XZ (Best compression)	0.0605237	3.88426	64.1775
XZ	0.030818	3.88426	126.039
ZIP	0.00768423	3.87759	504.617
7ZIP	0.0215967	3.87664	179.502
RAR	0.0236309	3.84245	162.603
LZ4 (Best compression)	0.00752068	2.94896	392.114
LZ4	0.00500178	2.94217	588.224

Test with "Artificial" sorted by "Time normalized compression"

Compression Algorithm	Time	Compression	Time normalized compression
LZ4	0.00500178	2.94217	588.224
PGZIP	0.00692916	3.89652	562.336
ZSTD	0.00734138	3.96104	539.549
ZIP	0.00768423	3.87759	504.617
PGZIP (Best compression)	0.00786114	3.89748	495.79
LZ4 (Best compression)	0.00752068	2.94896	392.114
7ZIP	0.0215967	3.87664	179.502
PLZIP	0.0232768	3.88643	166.966
RAR	0.0236309	3.84245	162.603
ZSTD (Best compression)	0.0301487	3.96554	131.532
XZ	0.030818	3.88426	126.039
LZMA	0.0315187	3.88753	123.341
PBZIP2	0.0370195	3.9214	105.928
PLZIP (Best compression)	0.040936	3.88733	94.9612
XZ (Best compression)	0.0605237	3.88426	64.1775
LZMA (Best compression)	0.0609977	3.88753	63.7324

Calgary

Test with "Calgary" sorted by "Time"

Compression Algorithm	Time	Compression	Time normalized compression
LZ4	0.0139112	1.928	138.593
PGZIP	0.0204134	3.04514	149.174
ZSTD	0.0300949	3.06354	101.796
PGZIP (Best compression)	0.0378704	3.06692	80.9846
PBZIP2	0.063678	3.65207	57.3522
RAR	0.0703645	3.3894	48.1692
ZIP	0.124563	3.03758	24.3858
LZ4 (Best compression)	0.220412	2.6397	11.9762
7ZIP	0.315797	3.79749	12.0251
XZ	0.733703	3.80797	5.19007
ZSTD (Best compression)	0.745826	3.63498	4.87377
LZMA	0.765039	3.80878	4.97855
XZ (Best compression)	0.783136	3.80797	4.86246
PLZIP	0.797309	3.79922	4.76505
LZMA (Best compression)	0.869179	3.80878	4.38205
PLZIP (Best compression)	1.05285	3.82562	3.63359

Test with "Calgary" sorted by "Compression"

Compression Algorithm	Time	Compression	Time normalized compression
PLZIP (Best compression)	1.05285	3.82562	3.63359
LZMA (Best compression)	0.869179	3.80878	4.38205
LZMA	0.765039	3.80878	4.97855
XZ (Best compression)	0.783136	3.80797	4.86246
XZ	0.733703	3.80797	5.19007
PLZIP	0.797309	3.79922	4.76505
7ZIP	0.315797	3.79749	12.0251
PBZIP2	0.063678	3.65207	57.3522
ZSTD (Best compression)	0.745826	3.63498	4.87377
RAR	0.0703645	3.3894	48.1692
PGZIP (Best compression)	0.0378704	3.06692	80.9846
ZSTD	0.0300949	3.06354	101.796
PGZIP	0.0204134	3.04514	149.174
ZIP	0.124563	3.03758	24.3858
LZ4 (Best compression)	0.220412	2.6397	11.9762
LZ4	0.0139112	1.928	138.593

Test with "Calgary" sorted by "Time normalized compression"

Compression Algorithm	Time	Compression	Time normalized compression
PGZIP	0.0204134	3.04514	149.174
LZ4	0.0139112	1.928	138.593
ZSTD	0.0300949	3.06354	101.796
PGZIP (Best compression)	0.0378704	3.06692	80.9846
PBZIP2	0.063678	3.65207	57.3522
RAR	0.0703645	3.3894	48.1692
ZIP	0.124563	3.03758	24.3858
7ZIP	0.315797	3.79749	12.0251
LZ4 (Best compression)	0.220412	2.6397	11.9762
XZ	0.733703	3.80797	5.19007
LZMA	0.765039	3.80878	4.97855
ZSTD (Best compression)	0.745826	3.63498	4.87377
XZ (Best compression)	0.783136	3.80797	4.86246
PLZIP	0.797309	3.79922	4.76505
LZMA (Best compression)	0.869179	3.80878	4.38205
PLZIP (Best compression)	1.05285	3.82562	3.63359

Canterbury

Test with "Canterbury" sorted by "Time"

Compression Algorithm	Time	Compression	Time normalized compression
LZ4	0.0113311	2.28631	201.773
PGZIP	0.0188959	3.82866	202.619
ZSTD	0.0193903	4.38005	225.888
PBZIP2	0.0492446	4.94642	100.446
RAR	0.0585382	5.45329	93.1579
PGZIP (Best compression)	0.0595274	3.84635	64.6147
ZIP	0.0982521	3.81731	38.8522
7ZIP	0.331636	5.78365	17.4398
XZ	0.630135	5.78572	9.18172
LZMA	0.638212	5.7872	9.06784
ZSTD (Best compression)	0.644854	5.45402	8.45777
XZ (Best compression)	0.658105	5.78572	8.79149
LZMA (Best compression)	0.674388	5.7872	8.58141
PLZIP	0.709754	5.77106	8.13107
LZ4 (Best compression)	0.981928	3.01902	3.07458
PLZIP (Best compression)	1.07809	5.83406	5.41146

Test with "Canterbury" sorted by "Compression"

Compression Algorithm	Time	Compression	Time normalized compression
PLZIP (Best compression)	1.07809	5.83406	5.41146
LZMA (Best compression)	0.674388	5.7872	8.58141
LZMA	0.638212	5.7872	9.06784
XZ (Best compression)	0.658105	5.78572	8.79149
XZ	0.630135	5.78572	9.18172
7ZIP	0.331636	5.78365	17.4398
PLZIP	0.709754	5.77106	8.13107
ZSTD (Best compression)	0.644854	5.45402	8.45777
RAR	0.0585382	5.45329	93.1579
PBZIP2	0.0492446	4.94642	100.446
ZSTD	0.0193903	4.38005	225.888
PGZIP (Best compression)	0.0595274	3.84635	64.6147
PGZIP	0.0188959	3.82866	202.619
ZIP	0.0982521	3.81731	38.8522
LZ4 (Best compression)	0.981928	3.01902	3.07458
LZ4	0.0113311	2.28631	201.773

Test with "Canterbury" sorted by "Time normalized compression"

Compression Algorithm	Time	Compression	Time normalized compression
ZSTD	0.0193903	4.38005	225.888
PGZIP	0.0188959	3.82866	202.619
LZ4	0.0113311	2.28631	201.773
PBZIP2	0.0492446	4.94642	100.446
RAR	0.0585382	5.45329	93.1579
PGZIP (Best compression)	0.0595274	3.84635	64.6147
ZIP	0.0982521	3.81731	38.8522
7ZIP	0.331636	5.78365	17.4398
XZ	0.630135	5.78572	9.18172
LZMA	0.638212	5.7872	9.06784
XZ (Best compression)	0.658105	5.78572	8.79149
LZMA (Best compression)	0.674388	5.7872	8.58141
ZSTD (Best compression)	0.644854	5.45402	8.45777
PLZIP	0.709754	5.77106	8.13107
PLZIP (Best compression)	1.07809	5.83406	5.41146
LZ4 (Best compression)	0.981928	3.01902	3.07458

Large

Test with "Large" sorted by "Time"

Compression Algorithm	Time	Compression	Time normalized compression
LZ4	0.0310037	1.92368	62.0468
ZSTD	0.0540023	3.45881	64.0493
PGZIP	0.0762684	3.42005	44.8423
PBZIP2	0.0938406	4.26873	45.4891
RAR	0.146149	3.94542	26.9958
PGZIP (Best compression)	0.204228	3.48357	17.0572
ZIP	0.935222	3.42537	3.66263
LZ4 (Best compression)	2.2331	2.95496	1.32326
7ZIP	2.33647	4.34037	1.85766
XZ	3.90453	4.36674	1.11838
ZSTD (Best compression)	3.95725	4.37968	1.10675
XZ (Best compression)	4.03401	4.36679	1.08249
LZMA	4.04633	4.3675	1.07938
LZMA (Best compression)	4.15799	4.36756	1.0504
PLZIP	4.66973	4.33882	0.929137
PLZIP (Best compression)	4.9531	4.36848	0.881968

Test with "Large" sorted by "Compression"

Compression Algorithm	Time	Compression	Time normalized compression
ZSTD (Best compression)	3.95725	4.37968	1.10675
PLZIP (Best compression)	4.9531	4.36848	0.881968
LZMA (Best compression)	4.15799	4.36756	1.0504
LZMA	4.04633	4.3675	1.07938
XZ (Best compression)	4.03401	4.36679	1.08249
XZ	3.90453	4.36674	1.11838
7ZIP	2.33647	4.34037	1.85766
PLZIP	4.66973	4.33882	0.929137
PBZIP2	0.0938406	4.26873	45.4891
RAR	0.146149	3.94542	26.9958
PGZIP (Best compression)	0.204228	3.48357	17.0572
ZSTD	0.0540023	3.45881	64.0493
ZIP	0.935222	3.42537	3.66263
PGZIP	0.0762684	3.42005	44.8423
LZ4 (Best compression)	2.2331	2.95496	1.32326
LZ4	0.0310037	1.92368	62.0468

Test with "Large" sorted by "Time normalized compression"

Compression Algorithm	Time	Compression	Time normalized compression
ZSTD	0.0540023	3.45881	64.0493
LZ4	0.0310037	1.92368	62.0468
PBZIP2	0.0938406	4.26873	45.4891
PGZIP	0.0762684	3.42005	44.8423
RAR	0.146149	3.94542	26.9958
PGZIP (Best compression)	0.204228	3.48357	17.0572
ZIP	0.935222	3.42537	3.66263
7ZIP	2.33647	4.34037	1.85766
LZ4 (Best compression)	2.2331	2.95496	1.32326
XZ	3.90453	4.36674	1.11838
ZSTD (Best compression)	3.95725	4.37968	1.10675
XZ (Best compression)	4.03401	4.36679	1.08249
LZMA	4.04633	4.3675	1.07938
LZMA (Best compression)	4.15799	4.36756	1.0504
PLZIP	4.66973	4.33882	0.929137
PLZIP (Best compression)	4.9531	4.36848	0.881968