From 4e09fa95cdb1fd4c9870cf92462e1c5868248510 Mon Sep 17 00:00:00 2001 From: Guilleag01 <51876317+Guilleag01@users.noreply.github.com> Date: Sun, 12 Jan 2025 13:59:25 +0100 Subject: [PATCH] Update README.md --- README.md | 290 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 288 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 382176d..99ed2dd 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,288 @@ -# compression_test -Some compression test on popular compression algorithms and tools +# Compression test + + +- [Compression test](#compression-test) + - [Test setup](#test-setup) + - [Results](#results) + - [Non compressible data](#non-compressible-data) + - [Lots of data with mixed types and sizes](#lots-of-data-with-mixed-types-and-sizes) + - [A lot of many small text files](#a-lot-of-many-small-text-files) + - [A few text files](#a-few-text-files) + - [A few mixed files](#a-few-mixed-files) + +I made some test on multiple of the compression algorithms and tools found on linux. + +## Test setup +The algorithms were tested on the following situations: + - **Lots of data with mixed types and sizes:** A lot of various types of files of varying sizes. The total size of all the files is 4.36 GiB + - **Non compressible data:** A lot of barely compressible files, in thes case jpeg images. The total size of all the files is 0.62 GiB + - **A lot of many small text files:** A lot small text files, all the files are under 1MB in size. The total size of all the files is 0.96 GiB + - **A few text files:** A few text files. The total size of all the files is 9.54 MiB + - **A few mixed files:** A few files varying in size and type. The total size of all the files is 6.66 MiB + +The algorithms tested were the following: + - XZ + - LZMA + - PGZIP (parallel gzip) + - PBZIP2 (parallel bzip2) + - LZ4 + - PLZIP (parallel lzip) + - ZSTD + - RAR + - 7ZIP + - ZIP + +The tests were run on a system with the following specs + - Ryzen 9 3900X CPU + - 32 GB DDR4 2133 MT/s memory + - nvme gen4 ssd + - Arch linux kernel version 6.12.8 + - xfs file system + +Notes: All the algorithms were run with the maximum compression mode. + +## Results +### Non compressible data + +Test with "Non compressible data" sorted by "Time" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| LZMA | 351.912 | 1.02573 | 0.00277034 | +| XZ | 104.294 | 1.02264 | 0.009376 | +| PLZIP | 37.1494 | 1.01846 | 0.0264305 | +| ZSTD | 26.1111 | 1.01446 | 0.0377519 | +| 7ZIP | 17.2523 | 1.01995 | 0.0568293 | +| ZIP | 16.3882 | 1.00645 | 0.0606285 | +| RAR | 13.2534 | 1.00414 | 0.0751413 | +| PBZIP2 | 3.8217 | 1.02804 | 0.254526 | +| LZ4 | 1.2858 | 0.999934 | 0.777779 | +| PGZIP | 1.26777 | 1.00505 | 0.784826 | + + +Test with "Non compressible data" sorted by "Compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| LZ4 | 1.2858 | 0.999934 | 0.777779 | +| RAR | 13.2534 | 1.00414 | 0.0751413 | +| PGZIP | 1.26777 | 1.00505 | 0.784826 | +| ZIP | 16.3882 | 1.00645 | 0.0606285 | +| ZSTD | 26.1111 | 1.01446 | 0.0377519 | +| PLZIP | 37.1494 | 1.01846 | 0.0264305 | +| 7ZIP | 17.2523 | 1.01995 | 0.0568293 | +| XZ | 104.294 | 1.02264 | 0.009376 | +| LZMA | 351.912 | 1.02573 | 0.00277034 | +| PBZIP2 | 3.8217 | 1.02804 | 0.254526 | + + +Test with "Non compressible data" sorted by "Time normalized compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| PGZIP | 1.26777 | 1.00505 | 0.784826 | +| LZ4 | 1.2858 | 0.999934 | 0.777779 | +| PBZIP2 | 3.8217 | 1.02804 | 0.254526 | +| RAR | 13.2534 | 1.00414 | 0.0751413 | +| ZIP | 16.3882 | 1.00645 | 0.0606285 | +| 7ZIP | 17.2523 | 1.01995 | 0.0568293 | +| ZSTD | 26.1111 | 1.01446 | 0.0377519 | +| PLZIP | 37.1494 | 1.01846 | 0.0264305 | +| XZ | 104.294 | 1.02264 | 0.009376 | +| LZMA | 351.912 | 1.02573 | 0.00277034 | + +### Lots of data with mixed types and sizes + +Test with "Lots of data with mixed types and sizes" sorted by "Time" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| LZMA | 1208.73 | 3.51061 | 0.000235662 | +| XZ | 236.184 | 3.46817 | 0.00122081 | +| ZSTD | 138.242 | 3.09079 | 0.0023404 | +| PLZIP | 135.211 | 3.343 | 0.00221234 | +| ZIP | 116.646 | 1.69067 | 0.00507074 | +| 7ZIP | 84.22 | 3.56446 | 0.00333112 | +| RAR | 70.6871 | 2.76305 | 0.00512 | +| LZ4 | 38.7711 | 1.61392 | 0.0159812 | +| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 | +| PGZIP | 21.8042 | 1.70312 | 0.0269288 | + + +Test with "Lots of data with mixed types and sizes" sorted by "Compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| LZ4 | 38.7711 | 1.61392 | 0.0159812 | +| ZIP | 116.646 | 1.69067 | 0.00507074 | +| PGZIP | 21.8042 | 1.70312 | 0.0269288 | +| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 | +| RAR | 70.6871 | 2.76305 | 0.00512 | +| ZSTD | 138.242 | 3.09079 | 0.0023404 | +| PLZIP | 135.211 | 3.343 | 0.00221234 | +| XZ | 236.184 | 3.46817 | 0.00122081 | +| LZMA | 1208.73 | 3.51061 | 0.000235662 | +| 7ZIP | 84.22 | 3.56446 | 0.00333112 | + + +Test with "Lots of data with mixed types and sizes" sorted by "Time normalized compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| PGZIP | 21.8042 | 1.70312 | 0.0269288 | +| PBZIP2 | 23.2807 | 2.08055 | 0.0206456 | +| LZ4 | 38.7711 | 1.61392 | 0.0159812 | +| RAR | 70.6871 | 2.76305 | 0.00512 | +| ZIP | 116.646 | 1.69067 | 0.00507074 | +| 7ZIP | 84.22 | 3.56446 | 0.00333112 | +| ZSTD | 138.242 | 3.09079 | 0.0023404 | +| PLZIP | 135.211 | 3.343 | 0.00221234 | +| XZ | 236.184 | 3.46817 | 0.00122081 | +| LZMA | 1208.73 | 3.51061 | 0.000235662 | + +### A lot of many small text files + +Test with "A lot of many small text files" sorted by "Time" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| LZMA | 380.253 | 7.65804 | 0.000343407 | +| XZ | 95.9468 | 7.61049 | 0.00136948 | +| PLZIP | 49.5483 | 7.60392 | 0.0026542 | +| ZSTD | 47.4384 | 7.19689 | 0.00292904 | +| 7ZIP | 46.3945 | 7.21744 | 0.00298642 | +| RAR | 34.9045 | 4.04528 | 0.00708224 | +| ZIP | 23.2824 | 3.73557 | 0.0114978 | +| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 | +| LZ4 | 4.07333 | 4.21619 | 0.0582278 | +| PGZIP | 3.20727 | 4.95067 | 0.0629797 | + + +Test with "A lot of many small text files" sorted by "Compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| ZIP | 23.2824 | 3.73557 | 0.0114978 | +| RAR | 34.9045 | 4.04528 | 0.00708224 | +| LZ4 | 4.07333 | 4.21619 | 0.0582278 | +| PGZIP | 3.20727 | 4.95067 | 0.0629797 | +| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 | +| ZSTD | 47.4384 | 7.19689 | 0.00292904 | +| 7ZIP | 46.3945 | 7.21744 | 0.00298642 | +| PLZIP | 49.5483 | 7.60392 | 0.0026542 | +| XZ | 95.9468 | 7.61049 | 0.00136948 | +| LZMA | 380.253 | 7.65804 | 0.000343407 | + + +Test with "A lot of many small text files" sorted by "Time normalized compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | ------: | ----------: | --------------------------: | +| PGZIP | 3.20727 | 4.95067 | 0.0629797 | +| LZ4 | 4.07333 | 4.21619 | 0.0582278 | +| PBZIP2 | 4.4668 | 6.37939 | 0.0350933 | +| ZIP | 23.2824 | 3.73557 | 0.0114978 | +| RAR | 34.9045 | 4.04528 | 0.00708224 | +| 7ZIP | 46.3945 | 7.21744 | 0.00298642 | +| ZSTD | 47.4384 | 7.19689 | 0.00292904 | +| PLZIP | 49.5483 | 7.60392 | 0.0026542 | +| XZ | 95.9468 | 7.61049 | 0.00136948 | +| LZMA | 380.253 | 7.65804 | 0.000343407 | + +### A few text files + +Test with "A few text files" sorted by "Time" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| PLZIP | 2.65034 | 2.14365 | 0.176013 | +| LZMA | 1.6571 | 2.14211 | 0.281714 | +| XZ | 1.64646 | 2.15047 | 0.282433 | +| ZSTD | 0.993853 | 2.15193 | 0.467573 | +| 7ZIP | 0.433445 | 2.15245 | 1.07185 | +| RAR | 0.322799 | 1.10265 | 2.80952 | +| PBZIP2 | 0.258476 | 1.37485 | 2.814 | +| ZIP | 0.241944 | 1.08497 | 3.80951 | +| LZ4 | 0.137808 | 1.13707 | 6.38176 | +| PGZIP | 0.0423446 | 1.13541 | 20.7993 | + + +Test with "A few text files" sorted by "Compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| ZIP | 0.241944 | 1.08497 | 3.80951 | +| RAR | 0.322799 | 1.10265 | 2.80952 | +| PGZIP | 0.0423446 | 1.13541 | 20.7993 | +| LZ4 | 0.137808 | 1.13707 | 6.38176 | +| PBZIP2 | 0.258476 | 1.37485 | 2.814 | +| LZMA | 1.6571 | 2.14211 | 0.281714 | +| PLZIP | 2.65034 | 2.14365 | 0.176013 | +| XZ | 1.64646 | 2.15047 | 0.282433 | +| ZSTD | 0.993853 | 2.15193 | 0.467573 | +| 7ZIP | 0.433445 | 2.15245 | 1.07185 | + + +Test with "A few text files" sorted by "Time normalized compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| PGZIP | 0.0423446 | 1.13541 | 20.7993 | +| LZ4 | 0.137808 | 1.13707 | 6.38176 | +| ZIP | 0.241944 | 1.08497 | 3.80951 | +| PBZIP2 | 0.258476 | 1.37485 | 2.814 | +| RAR | 0.322799 | 1.10265 | 2.80952 | +| 7ZIP | 0.433445 | 2.15245 | 1.07185 | +| ZSTD | 0.993853 | 2.15193 | 0.467573 | +| XZ | 1.64646 | 2.15047 | 0.282433 | +| LZMA | 1.6571 | 2.14211 | 0.281714 | +| PLZIP | 2.65034 | 2.14365 | 0.176013 | + +### A few mixed files + +Test with "A few mixed files" sorted by "Time" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| PLZIP | 2.20772 | 4.18295 | 0.108287 | +| ZSTD | 1.64639 | 3.75015 | 0.161965 | +| XZ | 1.56128 | 4.16868 | 0.153646 | +| LZMA | 1.36919 | 4.1687 | 0.1752 | +| 7ZIP | 0.53761 | 4.15235 | 0.447959 | +| ZIP | 0.240878 | 2.77821 | 1.4943 | +| RAR | 0.187353 | 3.48817 | 1.53018 | +| LZ4 | 0.178676 | 2.56754 | 2.17979 | +| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 | +| PGZIP | 0.0472157 | 2.8845 | 7.34247 | + + +Test with "A few mixed files" sorted by "Compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| LZ4 | 0.178676 | 2.56754 | 2.17979 | +| ZIP | 0.240878 | 2.77821 | 1.4943 | +| PGZIP | 0.0472157 | 2.8845 | 7.34247 | +| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 | +| RAR | 0.187353 | 3.48817 | 1.53018 | +| ZSTD | 1.64639 | 3.75015 | 0.161965 | +| 7ZIP | 0.53761 | 4.15235 | 0.447959 | +| XZ | 1.56128 | 4.16868 | 0.153646 | +| LZMA | 1.36919 | 4.1687 | 0.1752 | +| PLZIP | 2.20772 | 4.18295 | 0.108287 | + + +Test with "A few mixed files" sorted by "Time normalized compression" + +| Compression Algorithm | Time | Compression | Time normalized compression | +| :-------------------- | --------: | ----------: | --------------------------: | +| PGZIP | 0.0472157 | 2.8845 | 7.34247 | +| PBZIP2 | 0.0996904 | 3.3937 | 2.95579 | +| LZ4 | 0.178676 | 2.56754 | 2.17979 | +| RAR | 0.187353 | 3.48817 | 1.53018 | +| ZIP | 0.240878 | 2.77821 | 1.4943 | +| 7ZIP | 0.53761 | 4.15235 | 0.447959 | +| LZMA | 1.36919 | 4.1687 | 0.1752 | +| ZSTD | 1.64639 | 3.75015 | 0.161965 | +| XZ | 1.56128 | 4.16868 | 0.153646 | +| PLZIP | 2.20772 | 4.18295 | 0.108287 |