| erofs-utils |
| =========== |
| |
| erofs-utils includes user-space tools for erofs filesystem images. |
| Currently only mkfs.erofs is available. |
| |
| mkfs.erofs |
| ---------- |
| |
| It can create 2 primary kinds of erofs images: (un)compressed. |
| |
| - For compressed images, it's able to use several compression |
| algorithms, but lz4(hc) are only supported due to the current |
| linux kernel implementation. |
| |
| - For uncompressed images, it can decide whether the last page of |
| a file should be inlined or not properly [1]. |
| |
| Dependencies |
| ~~~~~~~~~~~~ |
| |
| lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.0+ recommended |
| |
| How to build for lz4-1.9.0 or above |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| To build you can run the following commands in order: |
| |
| :: |
| |
| $ ./autogen.sh |
| $ ./configure |
| $ make |
| |
| mkfs.erofs binary will be generated under mkfs folder. There are still |
| some issues which affect the stability of LZ4_compress_destSize() |
| * they have impacts on lz4 only rather than lz4HC * [3]. |
| |
| How to build for lz4-1.8.0~1.8.3 |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| For these old lz4 versions, lz4hc algorithm cannot be supported without |
| lz4 static libary due to LZ4_compress_HC_destSize unstable api usage, |
| which means only lz4 algrithm is available if lz4 static library isn't found. |
| |
| On Fedora, static lz4 can be installed using: |
| |
| yum install lz4-static.x86_64 |
| |
| However, it's not recommended to use those versions since there were bugs |
| in these compressors, see [2] [3] as well. |
| |
| How to generate erofs images |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Currently lz4 and lz4hc are available for compression, e.g. |
| $ mkfs.erofs -zlz4hc foo.erofs.img foo/ |
| |
| Or leave all files uncompressed as a option: |
| $ mkfs.erofs foo.erofs.img foo/ |
| |
| How to generate legacy erofs images |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Decompression inplace and compacted indexes have been introduced in |
| linux-5.3, which are not backward-compatible with older kernels. |
| |
| In order to generate _legacy_ erofs images for old kernels, |
| add "-E legacy-compress" to the command line, e.g. |
| |
| $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ |
| |
| |
| Known issues |
| ~~~~~~~~~~~~ |
| |
| 1. LZ4HC cannot compress long zeroed buffer properly with |
| LZ4_compress_HC_destSize() |
| https://github.com/lz4/lz4/issues/784 |
| |
| Obsoleted erofs.mkfs |
| ~~~~~~~~~~~~~~~~~~~~ |
| |
| There is an original erofs.mkfs version developped by Li Guifu, |
| which was replaced by the new erofs-utils implementation. |
| |
| git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs |
| |
| It may still be useful since new erofs-utils has not been widely used in |
| commercial products. However, if that happens, please report bug to us |
| as well. |
| |
| Contribution |
| ------------ |
| |
| erofs-utils is a GPLv2+ project as a part of erofs file system, |
| feel free to send patches or feedback to us. |
| |
| To: |
| linux-erofs mailing list <linux-erofs@lists.ozlabs.org> |
| Li Guifu <bluce.liguifu@huawei.com> |
| Miao Xie <miaoxie@huawei.com> |
| Fang Wei <fangwei1@huawei.com> |
| |
| Cc: |
| Gao Xiang <gaoxiang25@huawei.com> |
| Chao Yu <yuchao0@huawei.com> |
| |
| Comments |
| -------- |
| |
| [1] According to the erofs on-disk format, the last page of files could |
| be inlined aggressively with its metadata in order to reduce the I/O |
| overhead and save the storage space. |
| |
| [2] There was a bug until lz4-1.8.3, which can crash erofs-utils randomly. |
| Fortunately bugfix by our colleague Qiuyang Sun was merged in lz4-1.9.0. |
| |
| For more details, please refer to |
| https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 |
| |
| [3] There are many crash fixes merged to lz4 1.9.2 for LZ4_compress_destSize(), |
| and I once ran into some crashs due to those issues. |
| * Again lz4HC is not effected for this section. * |
| |
| [LZ4_compress_destSize] Allow 2 more bytes of match length |
| https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda |
| |
| [LZ4_compress_destSize] Fix rare data corruption bug |
| https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 |
| |
| [LZ4_compress_destSize] Fix overflow condition |
| https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a |
| |
| [LZ4_compress_destSize] Fix off-by-one error in fix |
| https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 |
| |
| [LZ4_compress_destSize] Fix off-by-one error |
| https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba |
| |
| since upstream lz4 doesn't have stable branch for old versions, it's |
| preferred to use latest upstream lz4 library (although some regressions |
| could happen since new features are also introduced to latest upstream |
| version as well) or backport all stable bugfixes to old stable versions, |
| e.g. our unoffical lz4 fork: https://github.com/erofs/lz4 |
| |
