orderfiles/scripts/README.md

Android Order Files Scripts
============================

For the latest version of this doc, please make sure to visit:
[Android Order Files Scripts](https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/main/orderfiles/scripts/README.md)

Getting started with Order files
----------------------------------
Order files are text files containing symbols representing functions names.
Linker (lld) uses order files to layout functions in a specific order.
These ordered binaries in Android will reduce page faults and improve a program's launch time due to the efficient loading of symbols during program’s cold-start.

The scripts described here are used to create and validate order files. You can learn how and when they are used by looking at [Android Order Files](https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/main/orderfiles/README.md).

File/CSV Format
----------------------------------
Some arguments in the script allows three formats (File, CSV, or Folder) based on the first character.
All formats represent a list of values, which is symbols or files in our case.
- File format: The file will have one value per line.
               Add @ before the filename to show it is a file.
               If the values are files, the format is (file, weight).
               Example: @example.txt
- CSV format: Use “” (Quotation) around the comma-separated values.
              Example: “main,foo,bar”
- Folder format: Add ^ before the path to the folder.
              We assume every file in the folder ends with ".orderfile".
              Example: ^path/to/folder

Orderfile scripts
----------------------------------
Following scripts are provided:
- [create_orderfile](create_orderfile.py)
- [validate_orderfile](validate_orderfile.py)
- [merge_orderfile](merge_orderfile.py)

In order to run the scripts you may need to install the following python3 dependencies:
- bitarray
- graphviz

Create Order file
----------------------------------
You can create an orderfile from a mapping file and profile file.

```
python3 create_orderfile [-h] --profile-file PROFILE_FILE --mapping-file MAPPING_FILE [--output OUTPUT] [--denylist DENYLIST] [--last-symbol LAST_SYMBOL] [--leftover]
```

Flags:
- Profile file (--profile-file):
    - Description: The profile file generated by running a binary compiled with forder-file-instrumentation
    - Type: String
    - Required
- Mapping file (--mapping-file):
    - Description: The mapping file generated during compilation that maps MD5 hashes to symbol names
    - Type: String
    - Required
- Output file (--output):
    - Description: The output file name for the order file. Default Name: default.orderfile
    - Type: String
- Deny List (--denylist):
    - Description: Symbols that you want to exclude from the order file
    - Type: String (File/CSV)
- Last symbol (--last-symbol):
    - Description: The order file will end at the passed last symbol and ignore the symbols after it.
                   If you want an order file only for startup, you should pass the last startup symbol.
                   Last-symbol has priority over leftover so we will output until the last symbol and ignore the leftover flag.
    - Type: String
- Leftover symbols (--leftover):
    - Description: Some symbols (functions) might not have executed so they will not appear in the profile file.
                   If you want these symbols in your orderfile, you can use this flag and it will add them at the end.
    - Type: Bool

Validate Order file
----------------------------------
Once we get an order file for a library or binary, we need to check if it is valid based on each team’s criteria.
To automate this process, we wrote a python script to check the criteria.
The current criteria that we allow:
- Defining an order priority that needs to be in the orderfile
- Symbols that have to be present in orderfile
- Symbols that should not be present in orderfile
- Minimum number of symbols to make an orderfile good for page layout purposes

```
python3 validate_orderfile [-h] --order-file ORDER_FILE [--partial PARTIAL] [--allowlist ALLOWLIST] [--denylist DENYLIST] [--min MIN]
```

Flags:
- Order file (--order-file):
    - Description: The order file that is being validated on the below criteria
    - Type: String
    - Required
- Partial Order (--partial):
    - Description: A partial order of symbols that must be correct in the order file.
    - Type: String (File/CSV)
- Allow List (--allowlist):
    - Description: Symbols that have to be present in orderfile
    - Type: String (File/CSV)
- Deny List (--denylist):
    - Description: Symbols that should not be present in orderfile. Denylist flag has priority over allowlist.
    - Type: String (File/CSV)
- Minimum Number of Entries (--min):
    - Description: Minimum number of symbols to make an orderfile good for page layout purposes
    - Type: Int

Merge Order File
----------------------------------
Any executable running on different devices might not create the same order file due to threads, OS, side effects, etc.
As a result, our script will take all the order files and merge them into one order file while trying to maintain locality.
As lower end device require better layout for performance boost, you can assign weights to order files and provide lower
end device order files with higher weight. You can only assign weights if you use File format and an example can be found
in test/merge-test/merge.txt.

```
python3 merge_orderfile [-h] --order-files ORDER_FILES [--output OUTPUT] [--graph-image GRAPH_IMAGE]
```

Flags:
- Files (--order-files):
    - Description: A collection of order files that need to be merged together
    - Type: String (File/CSV/Folder)
    - Required
- Output (--output):
    - Description: Provide the output file name for the order file. Default Name: default.orderfile
    - Type: String
- Graph Image (--graph-image):
    - Description: Provide the output image name for the graph representation of the order files
    - Type: String