Upgrade lz4 to v1.10.0 am: 5c55ae95f6

Original change: https://android-review.googlesource.com/c/platform/external/lz4/+/3288868

Change-Id: I11f34e35df7f9a5016044b9481492d3d272c79d8
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
tree: 85985f1627221aa31072e86fe6729e4990e92166
  1. .circleci/
  2. .github/
  3. build/
  4. contrib/
  5. doc/
  6. examples/
  7. lib/
  8. ossfuzz/
  9. programs/
  10. tests/
  11. .cirrus.yml
  12. .clang-format
  13. .gitattributes
  14. .gitignore
  15. appveyor.yml
  16. CODING_STYLE
  17. INSTALL
  18. LICENSE
  19. Makefile
  20. Makefile.inc
  21. METADATA
  22. MODULE_LICENSE_GPL
  23. NEWS
  24. OWNERS
  25. README.md
  26. SECURITY.md
README.md

LZ4 - Extremely fast compression

LZ4 is lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

Speed can be tuned dynamically, selecting an “acceleration” factor which trades compression ratio for faster speed. On the other end, a high compression derivative, LZ4_HC, is also provided, trading CPU time for improved compression ratio. All versions feature the same decompression speed.

LZ4 is also compatible with dictionary compression, both at API and CLI levels. It can ingest any input file as dictionary, though only the final 64KB are used. This capability can be combined with the Zstandard Dictionary Builder, in order to drastically improve compression performance on small files.

LZ4 library is provided as open-source software using BSD 2-Clause license.

BranchStatus
devBuild status

Benchmarks

The benchmark uses lzbench, from @inikep compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17). The reference system uses a Core i7-9700K CPU @ 4.9GHz (w/ turbo boost). Benchmark evaluates the compression of reference Silesia Corpus in single-thread mode.

CompressorRatioCompressionDecompression
memcpy1.00013700 MB/s13700 MB/s
LZ4 default (v1.9.0)2.101780 MB/s4970 MB/s
LZO 2.092.108670 MB/s860 MB/s
QuickLZ 1.5.02.238575 MB/s780 MB/s
Snappy 1.1.42.091565 MB/s1950 MB/s
Zstandard 1.4.0 -12.883515 MB/s1380 MB/s
LZF v3.62.073415 MB/s910 MB/s
zlib deflate 1.2.11 -12.730100 MB/s415 MB/s
LZ4 HC -9 (v1.9.0)2.72141 MB/s4900 MB/s
zlib deflate 1.2.11 -63.09936 MB/s445 MB/s

LZ4 is also compatible and optimized for x32 mode (-mx32), for which it provides additional speed performance.

Installation

make
make install     # this command may require root permissions

LZ4's Makefile supports standard Makefile conventions, including staged installs, redirection, or command redefinition. It is compatible with parallel builds (-j#).

Building LZ4 - Using vcpkg

You can download and install LZ4 using the vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg.exe install lz4

The LZ4 port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Documentation

The raw LZ4 block compression format is detailed within lz4_Block_format.

Arbitrarily long files or data streams are compressed using multiple blocks, for streaming requirements. These blocks are organized into a frame, defined into lz4_Frame_format. Interoperable versions of LZ4 must also respect the frame format.

Other source versions

Beyond the C reference source, many contributors have created versions of lz4 in multiple languages (Java, C#, Python, Perl, Ruby, etc.). A list of known source ports is maintained on the LZ4 Homepage.

Packaging status

Most distributions are bundled with a package manager which allows easy installation of both the liblz4 library and the lz4 command line interface.

Packaging status

Special Thanks

  • Takayuki Matsuoka, aka @t-mat, for exceptional first-class support throughout the lifetime of this project