guillermo/compression_test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c3df14afeb5d5fe0ccc508312e9d0dfe05660023
compression_test/README.md
Guilleag01 c3df14afeb Update README.md
2025-01-13 15:14:19 +01:00

343 lines
22 KiB
Markdown
Raw Blame History

# Compression test
- [Compression test](#compression-test)
- [Test setup](#test-setup)
- [Commands](#commands)
- [Results](#results)
- [Artificial](#artificial)
- [Calgary](#calgary)
- [Canterbury](#canterbury)
- [Large](#large)
I made some test on multiple of the compression algorithms and tools found on linux.
## Test setup
<!-- The algorithms were tested on the following situations:
- **Lots of data with mixed types and sizes:** A lot of various types of files of varying sizes. The total size of all the files is 4.36 GiB
- **Non-compressible data:** A lot of barely compressible files, in thes case jpeg images. The total size of all the files is 0.62 GiB
- **A lot of many small text files:** A lot of small text files, all the files are under 1MB in size. The total size of all the files is 0.96 GiB
- **A few text files:** A few text files. The total size of all the files is 9.54 MiB
- **A few mixed files:** A few files varying in size and type. The total size of all the files is 6.66 MiB -->
The algorithms have been tested with the [canterbury corpus](https://en.wikipedia.org/wiki/Canterbury_corpus). Downloaded from [this repository](https://github.com/pfalcon/canterbury-corpus).
The algorithms tested were the following:
- XZ
- LZMA
- PGZIP (parallel gzip)
- PBZIP2 (parallel bzip2)
- LZ4
- PLZIP (parallel lzip)
- ZSTD
- RAR
- 7ZIP
- ZIP
The tests were run on a system with the following specs
- Ryzen 9 3900X CPU
- 32 GB DDR4 2133 MT/s memory
- nvme gen4 ssd
- Arch linux kernel version 6.12.8
- xfs file system
Notes: Algorithms noyed with (Best compression) were run with the options for best compression otherwise default options are used.
### Commands
These were the commands used for each algorithm:
| Algorithm | Command |
| :----------------------- | :------------------------------------------ |
| 7ZIP | `7z a <archive> <dir>` |
| LZ4 | `tar c -I"lz4" -f <archive> <dir>` |
| LZ4 (Best compression) | `tar c -I"lz4 -12" -f <archive> <dir>` |
| LZMA | `tar c -I"lzma -T0" -f <archive> <dir>` |
| LZMA (Best compression) | `tar c -I"lzma -9 -T0" -f <archive> <dir>` |
| PBZIP2 | `tar c -Ipbzip2 -f <archive> <dir>` |
| PGZIP | `tar c -I"pigz" -f <archive> <dir>` |
| PGZIP (Best compression) | `tar c -I"pigz --best" -f <archive> <dir>` |
| PLZIP | `tar c -I"plzip" -f <archive> <dir>` |
| PLZIP (Best compression) | `tar c -I"plzip -9" -f <archive> <dir>` |
| RAR | `rar a <archive> <dir>` |
| XZ | `tar c -I"xz -T0" -f <archive> <dir>` |
| XZ (Best compression) | `tar c -I"xz -9 -T0" -f <archive> <dir>` |
| ZIP | `zip -r <archive> <dir>` |
| ZSTD | `tar c -I"zstd -T0" -f <archive> <dir>` |
| ZSTD (Best compression) | `tar c -I"zstd -19 -T0" -f <archive> <dir>` |
## Results
All the tables have the next categories:
- **Time:** The time the compression process took. Lower is better.
- **Compession:** The compression rate after the process. Higher is better.
- **Time normalized compression:** The compression normalized to the time obtained ny dividing the compression by the time. Higher is better.
### Artificial
Test with "Artificial" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | ---------: | ----------: | --------------------------: |
| LZ4 | 0.00500178 | 2.94217 | 588.224 |
| PGZIP | 0.00692916 | 3.89652 | 562.336 |
| ZSTD | 0.00734138 | 3.96104 | 539.549 |
| LZ4 (Best compression) | 0.00752068 | 2.94896 | 392.114 |
| ZIP | 0.00768423 | 3.87759 | 504.617 |
| PGZIP (Best compression) | 0.00786114 | 3.89748 | 495.79 |
| 7ZIP | 0.0215967 | 3.87664 | 179.502 |
| PLZIP | 0.0232768 | 3.88643 | 166.966 |
| RAR | 0.0236309 | 3.84245 | 162.603 |
| ZSTD (Best compression) | 0.0301487 | 3.96554 | 131.532 |
| XZ | 0.030818 | 3.88426 | 126.039 |
| LZMA | 0.0315187 | 3.88753 | 123.341 |
| PBZIP2 | 0.0370195 | 3.9214 | 105.928 |
| PLZIP (Best compression) | 0.040936 | 3.88733 | 94.9612 |
| XZ (Best compression) | 0.0605237 | 3.88426 | 64.1775 |
| LZMA (Best compression) | 0.0609977 | 3.88753 | 63.7324 |
Test with "Artificial" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | ---------: | ----------: | --------------------------: |
| ZSTD (Best compression) | 0.0301487 | 3.96554 | 131.532 |
| ZSTD | 0.00734138 | 3.96104 | 539.549 |
| PBZIP2 | 0.0370195 | 3.9214 | 105.928 |
| PGZIP (Best compression) | 0.00786114 | 3.89748 | 495.79 |
| PGZIP | 0.00692916 | 3.89652 | 562.336 |
| LZMA (Best compression) | 0.0609977 | 3.88753 | 63.7324 |
| LZMA | 0.0315187 | 3.88753 | 123.341 |
| PLZIP (Best compression) | 0.040936 | 3.88733 | 94.9612 |
| PLZIP | 0.0232768 | 3.88643 | 166.966 |
| XZ (Best compression) | 0.0605237 | 3.88426 | 64.1775 |
| XZ | 0.030818 | 3.88426 | 126.039 |
| ZIP | 0.00768423 | 3.87759 | 504.617 |
| 7ZIP | 0.0215967 | 3.87664 | 179.502 |
| RAR | 0.0236309 | 3.84245 | 162.603 |
| LZ4 (Best compression) | 0.00752068 | 2.94896 | 392.114 |
| LZ4 | 0.00500178 | 2.94217 | 588.224 |
Test with "Artificial" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | ---------: | ----------: | --------------------------: |
| LZ4 | 0.00500178 | 2.94217 | 588.224 |
| PGZIP | 0.00692916 | 3.89652 | 562.336 |
| ZSTD | 0.00734138 | 3.96104 | 539.549 |
| ZIP | 0.00768423 | 3.87759 | 504.617 |
| PGZIP (Best compression) | 0.00786114 | 3.89748 | 495.79 |
| LZ4 (Best compression) | 0.00752068 | 2.94896 | 392.114 |
| 7ZIP | 0.0215967 | 3.87664 | 179.502 |
| PLZIP | 0.0232768 | 3.88643 | 166.966 |
| RAR | 0.0236309 | 3.84245 | 162.603 |
| ZSTD (Best compression) | 0.0301487 | 3.96554 | 131.532 |
| XZ | 0.030818 | 3.88426 | 126.039 |
| LZMA | 0.0315187 | 3.88753 | 123.341 |
| PBZIP2 | 0.0370195 | 3.9214 | 105.928 |
| PLZIP (Best compression) | 0.040936 | 3.88733 | 94.9612 |
| XZ (Best compression) | 0.0605237 | 3.88426 | 64.1775 |
| LZMA (Best compression) | 0.0609977 | 3.88753 | 63.7324 |
### Calgary
Test with "Calgary" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.0139112 | 1.928 | 138.593 |
| PGZIP | 0.0204134 | 3.04514 | 149.174 |
| ZSTD | 0.0300949 | 3.06354 | 101.796 |
| PGZIP (Best compression) | 0.0378704 | 3.06692 | 80.9846 |
| PBZIP2 | 0.063678 | 3.65207 | 57.3522 |
| RAR | 0.0703645 | 3.3894 | 48.1692 |
| ZIP | 0.124563 | 3.03758 | 24.3858 |
| LZ4 (Best compression) | 0.220412 | 2.6397 | 11.9762 |
| 7ZIP | 0.315797 | 3.79749 | 12.0251 |
| XZ | 0.733703 | 3.80797 | 5.19007 |
| ZSTD (Best compression) | 0.745826 | 3.63498 | 4.87377 |
| LZMA | 0.765039 | 3.80878 | 4.97855 |
| XZ (Best compression) | 0.783136 | 3.80797 | 4.86246 |
| PLZIP | 0.797309 | 3.79922 | 4.76505 |
| LZMA (Best compression) | 0.869179 | 3.80878 | 4.38205 |
| PLZIP (Best compression) | 1.05285 | 3.82562 | 3.63359 |
Test with "Calgary" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| PLZIP (Best compression) | 1.05285 | 3.82562 | 3.63359 |
| LZMA (Best compression) | 0.869179 | 3.80878 | 4.38205 |
| LZMA | 0.765039 | 3.80878 | 4.97855 |
| XZ (Best compression) | 0.783136 | 3.80797 | 4.86246 |
| XZ | 0.733703 | 3.80797 | 5.19007 |
| PLZIP | 0.797309 | 3.79922 | 4.76505 |
| 7ZIP | 0.315797 | 3.79749 | 12.0251 |
| PBZIP2 | 0.063678 | 3.65207 | 57.3522 |
| ZSTD (Best compression) | 0.745826 | 3.63498 | 4.87377 |
| RAR | 0.0703645 | 3.3894 | 48.1692 |
| PGZIP (Best compression) | 0.0378704 | 3.06692 | 80.9846 |
| ZSTD | 0.0300949 | 3.06354 | 101.796 |
| PGZIP | 0.0204134 | 3.04514 | 149.174 |
| ZIP | 0.124563 | 3.03758 | 24.3858 |
| LZ4 (Best compression) | 0.220412 | 2.6397 | 11.9762 |
| LZ4 | 0.0139112 | 1.928 | 138.593 |
Test with "Calgary" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| PGZIP | 0.0204134 | 3.04514 | 149.174 |
| LZ4 | 0.0139112 | 1.928 | 138.593 |
| ZSTD | 0.0300949 | 3.06354 | 101.796 |
| PGZIP (Best compression) | 0.0378704 | 3.06692 | 80.9846 |
| PBZIP2 | 0.063678 | 3.65207 | 57.3522 |
| RAR | 0.0703645 | 3.3894 | 48.1692 |
| ZIP | 0.124563 | 3.03758 | 24.3858 |
| 7ZIP | 0.315797 | 3.79749 | 12.0251 |
| LZ4 (Best compression) | 0.220412 | 2.6397 | 11.9762 |
| XZ | 0.733703 | 3.80797 | 5.19007 |
| LZMA | 0.765039 | 3.80878 | 4.97855 |
| ZSTD (Best compression) | 0.745826 | 3.63498 | 4.87377 |
| XZ (Best compression) | 0.783136 | 3.80797 | 4.86246 |
| PLZIP | 0.797309 | 3.79922 | 4.76505 |
| LZMA (Best compression) | 0.869179 | 3.80878 | 4.38205 |
| PLZIP (Best compression) | 1.05285 | 3.82562 | 3.63359 |
### Canterbury
Test with "Canterbury" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.0113311 | 2.28631 | 201.773 |
| PGZIP | 0.0188959 | 3.82866 | 202.619 |
| ZSTD | 0.0193903 | 4.38005 | 225.888 |
| PBZIP2 | 0.0492446 | 4.94642 | 100.446 |
| RAR | 0.0585382 | 5.45329 | 93.1579 |
| PGZIP (Best compression) | 0.0595274 | 3.84635 | 64.6147 |
| ZIP | 0.0982521 | 3.81731 | 38.8522 |
| 7ZIP | 0.331636 | 5.78365 | 17.4398 |
| XZ | 0.630135 | 5.78572 | 9.18172 |
| LZMA | 0.638212 | 5.7872 | 9.06784 |
| ZSTD (Best compression) | 0.644854 | 5.45402 | 8.45777 |
| XZ (Best compression) | 0.658105 | 5.78572 | 8.79149 |
| LZMA (Best compression) | 0.674388 | 5.7872 | 8.58141 |
| PLZIP | 0.709754 | 5.77106 | 8.13107 |
| LZ4 (Best compression) | 0.981928 | 3.01902 | 3.07458 |
| PLZIP (Best compression) | 1.07809 | 5.83406 | 5.41146 |
Test with "Canterbury" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| PLZIP (Best compression) | 1.07809 | 5.83406 | 5.41146 |
| LZMA (Best compression) | 0.674388 | 5.7872 | 8.58141 |
| LZMA | 0.638212 | 5.7872 | 9.06784 |
| XZ (Best compression) | 0.658105 | 5.78572 | 8.79149 |
| XZ | 0.630135 | 5.78572 | 9.18172 |
| 7ZIP | 0.331636 | 5.78365 | 17.4398 |
| PLZIP | 0.709754 | 5.77106 | 8.13107 |
| ZSTD (Best compression) | 0.644854 | 5.45402 | 8.45777 |
| RAR | 0.0585382 | 5.45329 | 93.1579 |
| PBZIP2 | 0.0492446 | 4.94642 | 100.446 |
| ZSTD | 0.0193903 | 4.38005 | 225.888 |
| PGZIP (Best compression) | 0.0595274 | 3.84635 | 64.6147 |
| PGZIP | 0.0188959 | 3.82866 | 202.619 |
| ZIP | 0.0982521 | 3.81731 | 38.8522 |
| LZ4 (Best compression) | 0.981928 | 3.01902 | 3.07458 |
| LZ4 | 0.0113311 | 2.28631 | 201.773 |
Test with "Canterbury" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| ZSTD | 0.0193903 | 4.38005 | 225.888 |
| PGZIP | 0.0188959 | 3.82866 | 202.619 |
| LZ4 | 0.0113311 | 2.28631 | 201.773 |
| PBZIP2 | 0.0492446 | 4.94642 | 100.446 |
| RAR | 0.0585382 | 5.45329 | 93.1579 |
| PGZIP (Best compression) | 0.0595274 | 3.84635 | 64.6147 |
| ZIP | 0.0982521 | 3.81731 | 38.8522 |
| 7ZIP | 0.331636 | 5.78365 | 17.4398 |
| XZ | 0.630135 | 5.78572 | 9.18172 |
| LZMA | 0.638212 | 5.7872 | 9.06784 |
| XZ (Best compression) | 0.658105 | 5.78572 | 8.79149 |
| LZMA (Best compression) | 0.674388 | 5.7872 | 8.58141 |
| ZSTD (Best compression) | 0.644854 | 5.45402 | 8.45777 |
| PLZIP | 0.709754 | 5.77106 | 8.13107 |
| PLZIP (Best compression) | 1.07809 | 5.83406 | 5.41146 |
| LZ4 (Best compression) | 0.981928 | 3.01902 | 3.07458 |
### Large
Test with "Large" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.0310037 | 1.92368 | 62.0468 |
| ZSTD | 0.0540023 | 3.45881 | 64.0493 |
| PGZIP | 0.0762684 | 3.42005 | 44.8423 |
| PBZIP2 | 0.0938406 | 4.26873 | 45.4891 |
| RAR | 0.146149 | 3.94542 | 26.9958 |
| PGZIP (Best compression) | 0.204228 | 3.48357 | 17.0572 |
| ZIP | 0.935222 | 3.42537 | 3.66263 |
| LZ4 (Best compression) | 2.2331 | 2.95496 | 1.32326 |
| 7ZIP | 2.33647 | 4.34037 | 1.85766 |
| XZ | 3.90453 | 4.36674 | 1.11838 |
| ZSTD (Best compression) | 3.95725 | 4.37968 | 1.10675 |
| XZ (Best compression) | 4.03401 | 4.36679 | 1.08249 |
| LZMA | 4.04633 | 4.3675 | 1.07938 |
| LZMA (Best compression) | 4.15799 | 4.36756 | 1.0504 |
| PLZIP | 4.66973 | 4.33882 | 0.929137 |
| PLZIP (Best compression) | 4.9531 | 4.36848 | 0.881968 |
Test with "Large" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| ZSTD (Best compression) | 3.95725 | 4.37968 | 1.10675 |
| PLZIP (Best compression) | 4.9531 | 4.36848 | 0.881968 |
| LZMA (Best compression) | 4.15799 | 4.36756 | 1.0504 |
| LZMA | 4.04633 | 4.3675 | 1.07938 |
| XZ (Best compression) | 4.03401 | 4.36679 | 1.08249 |
| XZ | 3.90453 | 4.36674 | 1.11838 |
| 7ZIP | 2.33647 | 4.34037 | 1.85766 |
| PLZIP | 4.66973 | 4.33882 | 0.929137 |
| PBZIP2 | 0.0938406 | 4.26873 | 45.4891 |
| RAR | 0.146149 | 3.94542 | 26.9958 |
| PGZIP (Best compression) | 0.204228 | 3.48357 | 17.0572 |
| ZSTD | 0.0540023 | 3.45881 | 64.0493 |
| ZIP | 0.935222 | 3.42537 | 3.66263 |
| PGZIP | 0.0762684 | 3.42005 | 44.8423 |
| LZ4 (Best compression) | 2.2331 | 2.95496 | 1.32326 |
| LZ4 | 0.0310037 | 1.92368 | 62.0468 |
Test with "Large" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
| :----------------------- | --------: | ----------: | --------------------------: |
| ZSTD | 0.0540023 | 3.45881 | 64.0493 |
| LZ4 | 0.0310037 | 1.92368 | 62.0468 |
| PBZIP2 | 0.0938406 | 4.26873 | 45.4891 |
| PGZIP | 0.0762684 | 3.42005 | 44.8423 |
| RAR | 0.146149 | 3.94542 | 26.9958 |
| PGZIP (Best compression) | 0.204228 | 3.48357 | 17.0572 |
| ZIP | 0.935222 | 3.42537 | 3.66263 |
| 7ZIP | 2.33647 | 4.34037 | 1.85766 |
| LZ4 (Best compression) | 2.2331 | 2.95496 | 1.32326 |
| XZ | 3.90453 | 4.36674 | 1.11838 |
| ZSTD (Best compression) | 3.95725 | 4.37968 | 1.10675 |
| XZ (Best compression) | 4.03401 | 4.36679 | 1.08249 |
| LZMA | 4.04633 | 4.3675 | 1.07938 |
| LZMA (Best compression) | 4.15799 | 4.36756 | 1.0504 |
| PLZIP | 4.66973 | 4.33882 | 0.929137 |
| PLZIP (Best compression) | 4.9531 | 4.36848 | 0.881968 |
Reference in New Issue View Git Blame Copy Permalink