Compression test
I made some test on multiple of the compression algorithms and tools found on linux.
Test setup
The algorithms were tested on the following situations:
- Lots of data with mixed types and sizes: A lot of various types of files of varying sizes. The total size of all the files is 4.36 GiB
- Non-compressible data: A lot of barely compressible files, in thes case jpeg images. The total size of all the files is 0.62 GiB
- A lot of many small text files: A lot of small text files, all the files are under 1MB in size. The total size of all the files is 0.96 GiB
- A few text files: A few text files. The total size of all the files is 9.54 MiB
- A few mixed files: A few files varying in size and type. The total size of all the files is 6.66 MiB
The algorithms tested were the following:
- XZ
- LZMA
- PGZIP (parallel gzip)
- PBZIP2 (parallel bzip2)
- LZ4
- PLZIP (parallel lzip)
- ZSTD
- RAR
- 7ZIP
- ZIP
The tests were run on a system with the following specs
- Ryzen 9 3900X CPU
- 32 GB DDR4 2133 MT/s memory
- nvme gen4 ssd
- Arch linux kernel version 6.12.8
- xfs file system
Notes: All the algorithms were run with the maximum compression mode.
Results
Non-compressible data
Test with "Non-compressible data" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZMA | 351.912 | 1.02573 | 0.00277034 |
| XZ | 104.294 | 1.02264 | 0.009376 |
| PLZIP | 37.1494 | 1.01846 | 0.0264305 |
| ZSTD | 26.1111 | 1.01446 | 0.0377519 |
| 7ZIP | 17.2523 | 1.01995 | 0.0568293 |
| ZIP | 16.3882 | 1.00645 | 0.0606285 |
| RAR | 13.2534 | 1.00414 | 0.0751413 |
| PBZIP2 | 3.8217 | 1.02804 | 0.254526 |
| LZ4 | 1.2858 | 0.999934 | 0.777779 |
| PGZIP | 1.26777 | 1.00505 | 0.784826 |
Test with "Non-compressible data" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZ4 | 1.2858 | 0.999934 | 0.777779 |
| RAR | 13.2534 | 1.00414 | 0.0751413 |
| PGZIP | 1.26777 | 1.00505 | 0.784826 |
| ZIP | 16.3882 | 1.00645 | 0.0606285 |
| ZSTD | 26.1111 | 1.01446 | 0.0377519 |
| PLZIP | 37.1494 | 1.01846 | 0.0264305 |
| 7ZIP | 17.2523 | 1.01995 | 0.0568293 |
| XZ | 104.294 | 1.02264 | 0.009376 |
| LZMA | 351.912 | 1.02573 | 0.00277034 |
| PBZIP2 | 3.8217 | 1.02804 | 0.254526 |
Test with "Non-compressible data" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PGZIP | 1.26777 | 1.00505 | 0.784826 |
| LZ4 | 1.2858 | 0.999934 | 0.777779 |
| PBZIP2 | 3.8217 | 1.02804 | 0.254526 |
| RAR | 13.2534 | 1.00414 | 0.0751413 |
| ZIP | 16.3882 | 1.00645 | 0.0606285 |
| 7ZIP | 17.2523 | 1.01995 | 0.0568293 |
| ZSTD | 26.1111 | 1.01446 | 0.0377519 |
| PLZIP | 37.1494 | 1.01846 | 0.0264305 |
| XZ | 104.294 | 1.02264 | 0.009376 |
| LZMA | 351.912 | 1.02573 | 0.00277034 |
Lots of data with mixed types and sizes
Test with "Lots of data with mixed types and sizes" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZMA | 1208.73 | 3.51061 | 0.000235662 |
| XZ | 236.184 | 3.46817 | 0.00122081 |
| ZSTD | 138.242 | 3.09079 | 0.0023404 |
| PLZIP | 135.211 | 3.343 | 0.00221234 |
| ZIP | 116.646 | 1.69067 | 0.00507074 |
| 7ZIP | 84.22 | 3.56446 | 0.00333112 |
| RAR | 70.6871 | 2.76305 | 0.00512 |
| LZ4 | 38.7711 | 1.61392 | 0.0159812 |
| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 |
| PGZIP | 21.8042 | 1.70312 | 0.0269288 |
Test with "Lots of data with mixed types and sizes" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZ4 | 38.7711 | 1.61392 | 0.0159812 |
| ZIP | 116.646 | 1.69067 | 0.00507074 |
| PGZIP | 21.8042 | 1.70312 | 0.0269288 |
| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 |
| RAR | 70.6871 | 2.76305 | 0.00512 |
| ZSTD | 138.242 | 3.09079 | 0.0023404 |
| PLZIP | 135.211 | 3.343 | 0.00221234 |
| XZ | 236.184 | 3.46817 | 0.00122081 |
| LZMA | 1208.73 | 3.51061 | 0.000235662 |
| 7ZIP | 84.22 | 3.56446 | 0.00333112 |
Test with "Lots of data with mixed types and sizes" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PGZIP | 21.8042 | 1.70312 | 0.0269288 |
| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 |
| LZ4 | 38.7711 | 1.61392 | 0.0159812 |
| RAR | 70.6871 | 2.76305 | 0.00512 |
| ZIP | 116.646 | 1.69067 | 0.00507074 |
| 7ZIP | 84.22 | 3.56446 | 0.00333112 |
| ZSTD | 138.242 | 3.09079 | 0.0023404 |
| PLZIP | 135.211 | 3.343 | 0.00221234 |
| XZ | 236.184 | 3.46817 | 0.00122081 |
| LZMA | 1208.73 | 3.51061 | 0.000235662 |
A lot of many small text files
Test with "A lot of many small text files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZMA | 380.253 | 7.65804 | 0.000343407 |
| XZ | 95.9468 | 7.61049 | 0.00136948 |
| PLZIP | 49.5483 | 7.60392 | 0.0026542 |
| ZSTD | 47.4384 | 7.19689 | 0.00292904 |
| 7ZIP | 46.3945 | 7.21744 | 0.00298642 |
| RAR | 34.9045 | 4.04528 | 0.00708224 |
| ZIP | 23.2824 | 3.73557 | 0.0114978 |
| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 |
| LZ4 | 4.07333 | 4.21619 | 0.0582278 |
| PGZIP | 3.20727 | 4.95067 | 0.0629797 |
Test with "A lot of many small text files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| ZIP | 23.2824 | 3.73557 | 0.0114978 |
| RAR | 34.9045 | 4.04528 | 0.00708224 |
| LZ4 | 4.07333 | 4.21619 | 0.0582278 |
| PGZIP | 3.20727 | 4.95067 | 0.0629797 |
| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 |
| ZSTD | 47.4384 | 7.19689 | 0.00292904 |
| 7ZIP | 46.3945 | 7.21744 | 0.00298642 |
| PLZIP | 49.5483 | 7.60392 | 0.0026542 |
| XZ | 95.9468 | 7.61049 | 0.00136948 |
| LZMA | 380.253 | 7.65804 | 0.000343407 |
Test with "A lot of many small text files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PGZIP | 3.20727 | 4.95067 | 0.0629797 |
| LZ4 | 4.07333 | 4.21619 | 0.0582278 |
| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 |
| ZIP | 23.2824 | 3.73557 | 0.0114978 |
| RAR | 34.9045 | 4.04528 | 0.00708224 |
| 7ZIP | 46.3945 | 7.21744 | 0.00298642 |
| ZSTD | 47.4384 | 7.19689 | 0.00292904 |
| PLZIP | 49.5483 | 7.60392 | 0.0026542 |
| XZ | 95.9468 | 7.61049 | 0.00136948 |
| LZMA | 380.253 | 7.65804 | 0.000343407 |
A few text files
Test with "A few text files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PLZIP | 2.65034 | 2.14365 | 0.176013 |
| LZMA | 1.6571 | 2.14211 | 0.281714 |
| XZ | 1.64646 | 2.15047 | 0.282433 |
| ZSTD | 0.993853 | 2.15193 | 0.467573 |
| 7ZIP | 0.433445 | 2.15245 | 1.07185 |
| RAR | 0.322799 | 1.10265 | 2.80952 |
| PBZIP2 | 0.258476 | 1.37485 | 2.814 |
| ZIP | 0.241944 | 1.08497 | 3.80951 |
| LZ4 | 0.137808 | 1.13707 | 6.38176 |
| PGZIP | 0.0423446 | 1.13541 | 20.7993 |
Test with "A few text files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| ZIP | 0.241944 | 1.08497 | 3.80951 |
| RAR | 0.322799 | 1.10265 | 2.80952 |
| PGZIP | 0.0423446 | 1.13541 | 20.7993 |
| LZ4 | 0.137808 | 1.13707 | 6.38176 |
| PBZIP2 | 0.258476 | 1.37485 | 2.814 |
| LZMA | 1.6571 | 2.14211 | 0.281714 |
| PLZIP | 2.65034 | 2.14365 | 0.176013 |
| XZ | 1.64646 | 2.15047 | 0.282433 |
| ZSTD | 0.993853 | 2.15193 | 0.467573 |
| 7ZIP | 0.433445 | 2.15245 | 1.07185 |
Test with "A few text files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PGZIP | 0.0423446 | 1.13541 | 20.7993 |
| LZ4 | 0.137808 | 1.13707 | 6.38176 |
| ZIP | 0.241944 | 1.08497 | 3.80951 |
| PBZIP2 | 0.258476 | 1.37485 | 2.814 |
| RAR | 0.322799 | 1.10265 | 2.80952 |
| 7ZIP | 0.433445 | 2.15245 | 1.07185 |
| ZSTD | 0.993853 | 2.15193 | 0.467573 |
| XZ | 1.64646 | 2.15047 | 0.282433 |
| LZMA | 1.6571 | 2.14211 | 0.281714 |
| PLZIP | 2.65034 | 2.14365 | 0.176013 |
A few mixed files
Test with "A few mixed files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PLZIP | 2.20772 | 4.18295 | 0.108287 |
| ZSTD | 1.64639 | 3.75015 | 0.161965 |
| XZ | 1.56128 | 4.16868 | 0.153646 |
| LZMA | 1.36919 | 4.1687 | 0.1752 |
| 7ZIP | 0.53761 | 4.15235 | 0.447959 |
| ZIP | 0.240878 | 2.77821 | 1.4943 |
| RAR | 0.187353 | 3.48817 | 1.53018 |
| LZ4 | 0.178676 | 2.56754 | 2.17979 |
| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 |
| PGZIP | 0.0472157 | 2.8845 | 7.34247 |
Test with "A few mixed files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| LZ4 | 0.178676 | 2.56754 | 2.17979 |
| ZIP | 0.240878 | 2.77821 | 1.4943 |
| PGZIP | 0.0472157 | 2.8845 | 7.34247 |
| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 |
| RAR | 0.187353 | 3.48817 | 1.53018 |
| ZSTD | 1.64639 | 3.75015 | 0.161965 |
| 7ZIP | 0.53761 | 4.15235 | 0.447959 |
| XZ | 1.56128 | 4.16868 | 0.153646 |
| LZMA | 1.36919 | 4.1687 | 0.1752 |
| PLZIP | 2.20772 | 4.18295 | 0.108287 |
Test with "A few mixed files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|---|---|---|---|
| PGZIP | 0.0472157 | 2.8845 | 7.34247 |
| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 |
| LZ4 | 0.178676 | 2.56754 | 2.17979 |
| RAR | 0.187353 | 3.48817 | 1.53018 |
| ZIP | 0.240878 | 2.77821 | 1.4943 |
| 7ZIP | 0.53761 | 4.15235 | 0.447959 |
| LZMA | 1.36919 | 4.1687 | 0.1752 |
| ZSTD | 1.64639 | 3.75015 | 0.161965 |
| XZ | 1.56128 | 4.16868 | 0.153646 |
| PLZIP | 2.20772 | 4.18295 | 0.108287 |