| erofs-utils |
| =========== |
| |
| erofs-utils includes user-space tools for erofs filesystem. |
| Currently only mkfs.erofs is available. |
| |
| mkfs.erofs |
| ---------- |
| |
| It can generate 2 primary kinds of erofs images: (un)compressed. |
| |
| - For uncompressed images, there will be none of compression |
| files in these images. However, it can decide whether the tail |
| block of a file should be inlined or not properly [1]. |
| |
| - For compressed images, it will try to use lz4(hc) algorithm |
| first for each regular file and see if storage space can be |
| saved with compression. If not, fallback to an uncompressed |
| file. |
| |
| Dependencies |
| ~~~~~~~~~~~~ |
| |
| lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ highly recommended [4]. |
| |
| How to build for lz4-1.9.0 or above |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| To build, you can run the following commands in order: |
| |
| :: |
| |
| $ ./autogen.sh |
| $ ./configure |
| $ make |
| |
| mkfs.erofs binary will be generated under mkfs folder. |
| |
| * For lz4 < 1.9.2, there are some stability issues about |
| LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. |
| |
| How to build for lz4-1.8.0~1.8.3 |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| For these old lz4 versions, lz4hc algorithm cannot be supported |
| without lz4-static installed due to LZ4_compress_HC_destSize() |
| unstable api usage, which means lz4 will only be available if |
| lz4-static isn't found. |
| |
| On Fedora, lz4-static can be installed by using: |
| |
| yum install lz4-static.x86_64 |
| |
| However, it's still not recommended using those versions directly |
| since there are serious bugs in these compressors, see [2] [3] [4] |
| as well. |
| |
| How to generate erofs images |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Currently lz4 and lz4hc are available for compression, e.g. |
| $ mkfs.erofs -zlz4hc foo.erofs.img foo/ |
| |
| Or leave all files uncompressed as an option: |
| $ mkfs.erofs foo.erofs.img foo/ |
| |
| How to generate legacy erofs images |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Decompression inplace and compacted indexes have been introduced in |
| linux-5.3, which are not forward-compatible with older kernels. |
| |
| In order to generate _legacy_ erofs images for old kernels, |
| consider adding "-E legacy-compress" to the command line, e.g. |
| |
| $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ |
| |
| Obsoleted erofs.mkfs |
| ~~~~~~~~~~~~~~~~~~~~ |
| |
| There is an original erofs.mkfs version developed by Li Guifu, |
| which was replaced by the new erofs-utils implementation. |
| |
| git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs |
| |
| PLEASE NOTE: This version is highly _NOT recommended_ now. |
| |
| Contribution |
| ------------ |
| |
| erofs-utils is under GPLv2+ as a part of erofs project, |
| feel free to send patches or feedback to us. |
| |
| To: |
| linux-erofs mailing list <linux-erofs@lists.ozlabs.org> |
| Li Guifu <bluce.liguifu@huawei.com> |
| Miao Xie <miaoxie@huawei.com> |
| Fang Wei <fangwei1@huawei.com> |
| |
| Cc: |
| Gao Xiang <gaoxiang25@huawei.com> |
| Chao Yu <yuchao0@huawei.com> |
| |
| Comments |
| -------- |
| |
| [1] According to the erofs on-disk format, the tail block of files |
| could be inlined aggressively with its metadata in order to reduce |
| the I/O overhead and save the storage space (called tail-packing). |
| |
| [2] There was a bug until lz4-1.8.3, which can crash erofs-utils |
| randomly. Fortunately bugfix by our colleague Qiuyang Sun was |
| merged in lz4-1.9.0. |
| |
| For more details, please refer to |
| https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 |
| |
| [3] There were many bugfixes merged into lz4-1.9.2 for |
| LZ4_compress_destSize(), and I once ran into some crashs due to |
| those issues. * Again lz4hc is not affected. * |
| |
| [LZ4_compress_destSize] Allow 2 more bytes of match length |
| https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda |
| |
| [LZ4_compress_destSize] Fix rare data corruption bug |
| https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 |
| |
| [LZ4_compress_destSize] Fix overflow condition |
| https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a |
| |
| [LZ4_compress_destSize] Fix off-by-one error in fix |
| https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 |
| |
| [LZ4_compress_destSize] Fix off-by-one error |
| https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba |
| |
| since upstream lz4 doesn't have stable branch for old versions, it's |
| preferred to use latest upstream lz4 library (although some regressions |
| could happen since new features are also introduced to latest upstream |
| version as well) or backport all stable bugfixes to old stable versions, |
| e.g. our unofficial lz4 fork: https://github.com/erofs/lz4 |
| |
| [4] LZ4HC didn't compress long zeroed buffer properly with |
| LZ4_compress_HC_destSize() |
| https://github.com/lz4/lz4/issues/784 |
| |
| which has been resolved in |
| https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 |
| |
| and already included in lz4-1.9.3, see: |
| https://github.com/lz4/lz4/releases/tag/v1.9.3 |
| |
