README - platform/external/erofs-utils - Git at Google

 erofs-utils
 ===========

 erofs-utils includes user-space tools for EROFS filesystem.
 Currently mkfs.erofs, (experimental) erofsfuse, dump.erofs, fsck.erofs
 are available.

 Dependencies & build
 --------------------

  lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5].
  XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled.

  libfuse 2.6+ for erofsfuse enabled as a plus.

 How to build with lz4-1.9.0 or above
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 To build, you can run the following commands in order:

 ::

 	$ ./autogen.sh
 	$ ./configure
 	$ make

 mkfs.erofs binary will be generated under mkfs folder.

 * For lz4 < 1.9.2, there are some stability issues about
   LZ4_compress_destSize(). (lz4hc isn't impacted) [3].

 ** For lz4 = 1.9.2, there is a noticeable regression about
    LZ4_decompress_safe_partial() [5], which impacts erofsfuse
    functionality for legacy images (without 0PADDING).

 How to build with lz4-1.8.0~1.8.3
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 For these old lz4 versions, lz4hc algorithm cannot be supported
 without lz4-static installed due to LZ4_compress_HC_destSize()
 unstable api usage, which means lz4 will only be available if
 lz4-static isn't found.

 On Fedora, lz4-static can be installed by using:

 	yum install lz4-static.x86_64

 However, it's still not recommended using those versions directly
 since there are serious bugs in these compressors, see [2] [3] [4]
 as well.

 How to build with liblzma
 ~~~~~~~~~~~~~~~~~~~~~~~~~

 In order to enable LZMA support, build with the following commands:
 	$ ./configure --enable-lzma
 	$ make

 Additionally, you could specify liblzma build paths with:
 	--with-liblzma-incdir and --with-liblzma-libdir

 mkfs.erofs
 ----------

 two main kinds of EROFS images can be generated: (un)compressed.

  - For uncompressed images, there will be none of compression
    files in these images. However, it can decide whether the tail
    block of a file should be inlined or not properly [1].

  - For compressed images, it'll try to use specific algorithms
    first for each regular file and see if storage space can be
    saved with compression. If not, fallback to an uncompressed
    file.

 How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Currently lz4(hc) and lzma are available for compression, e.g.
  $ mkfs.erofs -zlz4hc foo.erofs.img foo/

 Or leave all files uncompressed as an option:
  $ mkfs.erofs foo.erofs.img foo/

 In addition, you could specify a higher compression level to get a
 (slightly) better compression ratio than the default level, e.g.
  $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/

 Note that all compressors are still single-threaded for now, thus it
 could take more time on the multiprocessor platform. Multi-threaded
 approach is already in our TODO list.

 How to generate EROFS big pcluster images (Linux 5.13+)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 In order to get much better compression ratios (thus better sequential
 read performance for common storage devices), big pluster feature has
 been introduced since linux-5.13, which is not forward-compatible with
 old kernels.

 In details, -C is used to specify the maximum size of each big pcluster
 in bytes, e.g.
  $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/

 So in that case, pcluster size can be 64KiB at most.

 Note that large pcluster size can cause bad random performance, so
 please evaluate carefully in advance. Or make your own per-(sub)file
 compression strategies according to file access patterns if needed.

 How to generate legacy EROFS images (Linux 4.19+)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Decompression inplace and compacted indexes have been introduced in
 Linux upstream v5.3, which are not forward-compatible with older
 kernels.

 In order to generate _legacy_ EROFS images for old kernels,
 consider adding "-E legacy-compress" to the command line, e.g.

  $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/

 For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_
 due to runtime performance loss compared with non-legacy images.

 Obsoleted erofs.mkfs
 ~~~~~~~~~~~~~~~~~~~~

 There is an original erofs.mkfs version developed by Li Guifu,
 which was replaced by the new erofs-utils implementation.

 git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs

 PLEASE NOTE: This version is highly _NOT recommended_ now.

 erofsfuse (experimental)
 ------------------------

 erofsfuse is introduced to support EROFS format for various platforms
 (including older linux kernels) and new on-disk features iteration.
 It can also be used as an unpacking tool for unprivileged users.

 It supports fixed-sized output decompression *without* any in-place
 I/O or in-place decompression optimization. Also like the other FUSE
 implementations, it suffers from most common performance issues (e.g.
 significant I/O overhead, double caching, etc.)

 Therefore, NEVER use it if performance is the top concern.

 Note that xattr & ACL aren't implemented yet due to the current Android
 use-case vs limited time. If you have some interest, contribution is,
 as always, welcome.

 How to build erofsfuse
 ~~~~~~~~~~~~~~~~~~~~~~

 It's disabled by default as an experimental feature for now due to
 the extra libfuse dependency, to enable and build it manually:

 	$ ./configure --enable-fuse
 	$ make

 erofsfuse binary will be generated under fuse folder.

 How to mount an EROFS image with erofsfuse
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 As the other FUSE implementations, it's quite simple to mount with
 erofsfuse, e.g.:
  $ erofsfuse foo.erofs.img foo/

 Alternatively, to make it run in foreground (with debugging level 3):
  $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/

 To debug erofsfuse (also automatically run in foreground):
  $ erofsfuse -d foo.erofs.img foo/

 To unmount an erofsfuse mountpoint as a non-root user:
  $ fusermount -u foo/

 dump.erofs and fsck.erofs (experimental)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 dump.erofs and fsck.erofs are two new experimental tools to analyse
 and check EROFS file systems.

 They are still incomplete and actively under development by the
 community. But you could check them out if needed in advance.

 Report, feedback and/or contribution are welcomed.

 Contribution
 ------------

 erofs-utils is under GPLv2+ as a part of EROFS filesystem project,
 feel free to send patches or feedback to:
   linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>

 Comments
 --------

 [1] According to the EROFS on-disk format, the tail block of files
     could be inlined aggressively with its metadata in order to reduce
     the I/O overhead and save the storage space (called tail-packing).

 [2] There was a bug until lz4-1.8.3, which can crash erofs-utils
     randomly. Fortunately bugfix by our colleague Qiuyang Sun was
     merged in lz4-1.9.0.

     For more details, please refer to
     https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82

 [3] There were many bugfixes merged into lz4-1.9.2 for
     LZ4_compress_destSize(), and I once ran into some crashs due to
     those issues. * Again lz4hc is not affected. *

     [LZ4_compress_destSize] Allow 2 more bytes of match length
     https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda

     [LZ4_compress_destSize] Fix rare data corruption bug
     https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1

     [LZ4_compress_destSize] Fix overflow condition
     https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a

     [LZ4_compress_destSize] Fix off-by-one error in fix
     https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333

     [LZ4_compress_destSize] Fix off-by-one error
     https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba

     since upstream lz4 doesn't have stable branch for old versions, it's
     preferred to use latest upstream lz4 library (although some regressions
     could happen since new features are also introduced to latest upstream
     version as well) or backport all stable bugfixes to old stable versions,
     e.g. our unofficial lz4 fork: https://github.com/erofs/lz4

 [4] LZ4HC didn't compress long zeroed buffer properly with
     LZ4_compress_HC_destSize()
     https://github.com/lz4/lz4/issues/784

     which has been resolved in
     https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1

     and already included in lz4-1.9.3, see:
     https://github.com/lz4/lz4/releases/tag/v1.9.3

 [5] LZ4_decompress_safe_partial is broken in 1.9.2
     https://github.com/lz4/lz4/issues/783

     which is also resolved in lz4-1.9.3.

 [6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz
	erofs-utils
	===========

	erofs-utils includes user-space tools for EROFS filesystem.
	Currently mkfs.erofs, (experimental) erofsfuse, dump.erofs, fsck.erofs
	are available.

	Dependencies & build
	--------------------

	lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5].
	XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled.

	libfuse 2.6+ for erofsfuse enabled as a plus.

	How to build with lz4-1.9.0 or above
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	To build, you can run the following commands in order:

	::

	$ ./autogen.sh
	$ ./configure
	$ make

	mkfs.erofs binary will be generated under mkfs folder.

	* For lz4 < 1.9.2, there are some stability issues about
	LZ4_compress_destSize(). (lz4hc isn't impacted) [3].

	** For lz4 = 1.9.2, there is a noticeable regression about
	LZ4_decompress_safe_partial() [5], which impacts erofsfuse
	functionality for legacy images (without 0PADDING).

	How to build with lz4-1.8.0~1.8.3
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	For these old lz4 versions, lz4hc algorithm cannot be supported
	without lz4-static installed due to LZ4_compress_HC_destSize()
	unstable api usage, which means lz4 will only be available if
	lz4-static isn't found.

	On Fedora, lz4-static can be installed by using:

	yum install lz4-static.x86_64

	However, it's still not recommended using those versions directly
	since there are serious bugs in these compressors, see [2] [3] [4]
	as well.

	How to build with liblzma
	~~~~~~~~~~~~~~~~~~~~~~~~~

	In order to enable LZMA support, build with the following commands:
	$ ./configure --enable-lzma
	$ make

	Additionally, you could specify liblzma build paths with:
	--with-liblzma-incdir and --with-liblzma-libdir

	mkfs.erofs
	----------

	two main kinds of EROFS images can be generated: (un)compressed.

	- For uncompressed images, there will be none of compression
	files in these images. However, it can decide whether the tail
	block of a file should be inlined or not properly [1].

	- For compressed images, it'll try to use specific algorithms
	first for each regular file and see if storage space can be
	saved with compression. If not, fallback to an uncompressed
	file.

	How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Currently lz4(hc) and lzma are available for compression, e.g.
	$ mkfs.erofs -zlz4hc foo.erofs.img foo/

	Or leave all files uncompressed as an option:
	$ mkfs.erofs foo.erofs.img foo/

	In addition, you could specify a higher compression level to get a
	(slightly) better compression ratio than the default level, e.g.
	$ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/

	Note that all compressors are still single-threaded for now, thus it
	could take more time on the multiprocessor platform. Multi-threaded
	approach is already in our TODO list.

	How to generate EROFS big pcluster images (Linux 5.13+)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	In order to get much better compression ratios (thus better sequential
	read performance for common storage devices), big pluster feature has
	been introduced since linux-5.13, which is not forward-compatible with
	old kernels.

	In details, -C is used to specify the maximum size of each big pcluster
	in bytes, e.g.
	$ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/

	So in that case, pcluster size can be 64KiB at most.

	Note that large pcluster size can cause bad random performance, so
	please evaluate carefully in advance. Or make your own per-(sub)file
	compression strategies according to file access patterns if needed.

	How to generate legacy EROFS images (Linux 4.19+)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Decompression inplace and compacted indexes have been introduced in
	Linux upstream v5.3, which are not forward-compatible with older
	kernels.

	In order to generate _legacy_ EROFS images for old kernels,
	consider adding "-E legacy-compress" to the command line, e.g.

	$ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/

	For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_
	due to runtime performance loss compared with non-legacy images.

	Obsoleted erofs.mkfs
	~~~~~~~~~~~~~~~~~~~~

	There is an original erofs.mkfs version developed by Li Guifu,
	which was replaced by the new erofs-utils implementation.

	git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs

	PLEASE NOTE: This version is highly _NOT recommended_ now.

	erofsfuse (experimental)
	------------------------

	erofsfuse is introduced to support EROFS format for various platforms
	(including older linux kernels) and new on-disk features iteration.
	It can also be used as an unpacking tool for unprivileged users.

	It supports fixed-sized output decompression without any in-place
	I/O or in-place decompression optimization. Also like the other FUSE
	implementations, it suffers from most common performance issues (e.g.
	significant I/O overhead, double caching, etc.)

	Therefore, NEVER use it if performance is the top concern.

	Note that xattr & ACL aren't implemented yet due to the current Android
	use-case vs limited time. If you have some interest, contribution is,
	as always, welcome.

	How to build erofsfuse
	~~~~~~~~~~~~~~~~~~~~~~

	It's disabled by default as an experimental feature for now due to
	the extra libfuse dependency, to enable and build it manually:

	$ ./configure --enable-fuse
	$ make

	erofsfuse binary will be generated under fuse folder.

	How to mount an EROFS image with erofsfuse
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	As the other FUSE implementations, it's quite simple to mount with
	erofsfuse, e.g.:
	$ erofsfuse foo.erofs.img foo/

	Alternatively, to make it run in foreground (with debugging level 3):
	$ erofsfuse -f --dbglevel=3 foo.erofs.img foo/

	To debug erofsfuse (also automatically run in foreground):
	$ erofsfuse -d foo.erofs.img foo/

	To unmount an erofsfuse mountpoint as a non-root user:
	$ fusermount -u foo/

	dump.erofs and fsck.erofs (experimental)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	dump.erofs and fsck.erofs are two new experimental tools to analyse
	and check EROFS file systems.

	They are still incomplete and actively under development by the
	community. But you could check them out if needed in advance.

	Report, feedback and/or contribution are welcomed.

	Contribution
	------------

	erofs-utils is under GPLv2+ as a part of EROFS filesystem project,
	feel free to send patches or feedback to:
	linux-erofs mailing list <linux-erofs@lists.ozlabs.org>

	Comments
	--------

	[1] According to the EROFS on-disk format, the tail block of files
	could be inlined aggressively with its metadata in order to reduce
	the I/O overhead and save the storage space (called tail-packing).

	[2] There was a bug until lz4-1.8.3, which can crash erofs-utils
	randomly. Fortunately bugfix by our colleague Qiuyang Sun was
	merged in lz4-1.9.0.

	For more details, please refer to
	https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82

	[3] There were many bugfixes merged into lz4-1.9.2 for
	LZ4_compress_destSize(), and I once ran into some crashs due to
	those issues. * Again lz4hc is not affected. *

	[LZ4_compress_destSize] Allow 2 more bytes of match length
	https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda

	[LZ4_compress_destSize] Fix rare data corruption bug
	https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1

	[LZ4_compress_destSize] Fix overflow condition
	https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a

	[LZ4_compress_destSize] Fix off-by-one error in fix
	https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333

	[LZ4_compress_destSize] Fix off-by-one error
	https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba

	since upstream lz4 doesn't have stable branch for old versions, it's
	preferred to use latest upstream lz4 library (although some regressions
	could happen since new features are also introduced to latest upstream
	version as well) or backport all stable bugfixes to old stable versions,
	e.g. our unofficial lz4 fork: https://github.com/erofs/lz4

	[4] LZ4HC didn't compress long zeroed buffer properly with
	LZ4_compress_HC_destSize()
	https://github.com/lz4/lz4/issues/784

	which has been resolved in
	https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1

	and already included in lz4-1.9.3, see:
	https://github.com/lz4/lz4/releases/tag/v1.9.3

	[5] LZ4_decompress_safe_partial is broken in 1.9.2
	https://github.com/lz4/lz4/issues/783

	which is also resolved in lz4-1.9.3.

	[6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz