guillermo/compression_test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Guilleag01
2025-01-12 16:00:44 +01:00
committed by GitHub
parent 1a58735e47
commit 57a254dfc5
1 changed files with 320 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
README.md
View File

@@ -1,9 +1,59 @@
# Compression test
- [Compression test](#compression-test)
- [Test setup](#test-setup)
- [Results](#results)
- [Non compressible data](#non-compressible-data)
- [Lots of data with mixed types and sizes](#lots-of-data-with-mixed-types-and-sizes)
- [A lot of many small text files](#a-lot-of-many-small-text-files)
- [A few text files](#a-few-text-files)
- [A few mixed files](#a-few-mixed-files)
I made some test on multiple of the compression algorithms and tools found on linux.
## Test setup
The algorithms were tested on the following situations:
- **Lots of data with mixed types and sizes:** A lot of various types of files of varying sizes. The total size of all the files is 4.36 GiB
- **Non-compressible data:** A lot of barely compressible files, in thes case jpeg images. The total size of all the files is 0.62 GiB
- **A lot of many small text files:** A lot of small text files, all the files are under 1MB in size. The total size of all the files is 0.96 GiB
- **A few text files:** A few text files. The total size of all the files is 9.54 MiB
- **A few mixed files:** A few files varying in size and type. The total size of all the files is 6.66 MiB
The algorithms tested were the following:
- XZ
- LZMA
- PGZIP (parallel gzip)
- PBZIP2 (parallel bzip2)
- LZ4
- PLZIP (parallel lzip)
- ZSTD
- RAR
- 7ZIP
- ZIP
The tests were run on a system with the following specs
- Ryzen 9 3900X CPU
- 32 GB DDR4 2133 MT/s memory
- nvme gen4 ssd
- Arch linux kernel version 6.12.8
- xfs file system
Notes: Algorithms noyed with (Best compression) were run with the options for best compression otherwise default options are used.
## Results
All the tables have the next categories:
- **Time:** The time the compression process took. Lower is better.
- **Compession:** The compression rate after the process. Higher is better.
- **Time normalized compression:** The compression normalized to the time obtained ny dividing the compression by the time. Higher is better.
### Non compressible data
Test with "Non compressible data" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | -------: | ----------: | --------------------------: |
| ZSTD | 0.618463 | 1.00274 | 1.6125 |
| LZ4 | 0.674424 | 0.999934 | 1.48284 |
| LZ4 (Best compression) | 1.31029 | 0.999934 | 0.76324 |
@@ -25,7 +75,7 @@ Test with "Non compressible data" sorted by "Time"
Test with "Non compressible data" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | -------: | ----------: | --------------------------: |
| PBZIP2 | 4.0795 | 1.02804 | 0.238442 |
| LZMA (Best compression) | 355.775 | 1.02573 | 0.00274026 |
| XZ (Best compression) | 105.435 | 1.02264 | 0.00927452 |
@@ -47,7 +97,7 @@ Test with "Non compressible data" sorted by "Compression"
Test with "Non compressible data" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | -------: | ----------: | --------------------------: |
| ZSTD | 0.618463 | 1.00274 | 1.6125 |
| LZ4 | 0.674424 | 0.999934 | 1.48284 |
| LZ4 (Best compression) | 1.31029 | 0.999934 | 0.76324 |
@@ -70,7 +120,7 @@ Test with "Non compressible data" sorted by "Time normalized compression"
Test with "Lots of data with mixed types and sizes" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| ZSTD | 3.46664 | 2.51686 | 0.114613 |
| LZ4 | 4.21573 | 1.5038 | 0.157738 |
| PGZIP | 9.69945 | 1.6994 | 0.0606677 |
@@ -92,7 +142,7 @@ Test with "Lots of data with mixed types and sizes" sorted by "Time"
Test with "Lots of data with mixed types and sizes" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| 7ZIP | 95.7274 | 3.56446 | 0.00293069 |
| LZMA (Best compression) | 1226.16 | 3.51061 | 0.000232311 |
| XZ (Best compression) | 240.062 | 3.46817 | 0.00120109 |
@@ -114,7 +164,7 @@ Test with "Lots of data with mixed types and sizes" sorted by "Compression"
Test with "Lots of data with mixed types and sizes" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|-----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| LZ4 | 4.21573 | 1.5038 | 0.157738 |
| ZSTD | 3.46664 | 2.51686 | 0.114613 |
| PGZIP | 9.69945 | 1.6994 | 0.0606677 |
@@ -137,7 +187,7 @@ Test with "Lots of data with mixed types and sizes" sorted by "Time normalized c
Test with "A lot of many small text files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| ZSTD | 1.18429 | 5.13139 | 0.164554 |
| LZ4 | 1.51286 | 3.02663 | 0.218395 |
| PGZIP | 2.15661 | 4.89074 | 0.0948099 |
@@ -159,7 +209,7 @@ Test with "A lot of many small text files" sorted by "Time"
Test with "A lot of many small text files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| LZMA (Best compression) | 401.785 | 7.65804 | 0.000325004 |
| XZ (Best compression) | 101.392 | 7.61049 | 0.00129593 |
| PLZIP (Best compression) | 53.0104 | 7.60392 | 0.00248085 |
@@ -181,7 +231,7 @@ Test with "A lot of many small text files" sorted by "Compression"
Test with "A lot of many small text files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | ------: | ----------: | --------------------------: |
| LZ4 | 1.51286 | 3.02663 | 0.218395 |
| ZSTD | 1.18429 | 5.13139 | 0.164554 |
| PGZIP | 2.15661 | 4.89074 | 0.0948099 |
@@ -204,7 +254,7 @@ Test with "A lot of many small text files" sorted by "Time normalized compressio
Test with "A few text files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.023663 | 1.1127 | 37.9798 |
| PGZIP | 0.0332351 | 1.13478 | 26.5149 |
| PGZIP (Best compression) | 0.0341132 | 1.13541 | 25.818 |
@@ -226,7 +276,7 @@ Test with "A few text files" sorted by "Time"
Test with "A few text files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| 7ZIP | 0.426743 | 2.15245 | 1.08868 |
| ZSTD (Best compression) | 0.928809 | 2.15193 | 0.500317 |
| XZ (Best compression) | 1.55916 | 2.15047 | 0.298247 |
@@ -248,7 +298,7 @@ Test with "A few text files" sorted by "Compression"
Test with "A few text files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.023663 | 1.1127 | 37.9798 |
| PGZIP | 0.0332351 | 1.13478 | 26.5149 |
| PGZIP (Best compression) | 0.0341132 | 1.13541 | 25.818 |
@@ -271,7 +321,7 @@ Test with "A few text files" sorted by "Time normalized compression"
Test with "A few mixed files" sorted by "Time"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.0194356 | 1.99491 | 25.7916 |
| PGZIP | 0.0281055 | 2.8725 | 12.3865 |
| ZSTD | 0.0360215 | 3.09782 | 8.96153 |
@@ -293,7 +343,7 @@ Test with "A few mixed files" sorted by "Time"
Test with "A few mixed files" sorted by "Compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| PLZIP (Best compression) | 2.02732 | 4.18295 | 0.117922 |
| LZMA (Best compression) | 1.29071 | 4.1687 | 0.185853 |
| LZMA | 1.23839 | 4.1687 | 0.193706 |
@@ -315,7 +365,7 @@ Test with "A few mixed files" sorted by "Compression"
Test with "A few mixed files" sorted by "Time normalized compression"
| Compression Algorithm | Time | Compression | Time normalized compression |
|:-------------------------|----------:|--------------:|------------------------------:|
| :----------------------- | --------: | ----------: | --------------------------: |
| LZ4 | 0.0194356 | 1.99491 | 25.7916 |
| PGZIP | 0.0281055 | 2.8725 | 12.3865 |
| ZSTD | 0.0360215 | 3.09782 | 8.96153 |