A new stable version of Rust is released approximately every 6 weeks. This document describes the process to update the Android Rust toolchain for the latest Stable Rust release. Other Googler developers use the artifacts produced as a result of this process.
In order to resolve issues caused Rust toolchain updates ahead of time we will perform the release process on the Beta version of the toolchain. Testing against the Beta version of the toolchain can often hint at possible incompatibilities between the Rust toolchain and the Android code base. However, sometimes there will be language features available in a Beta that have not been released in the Stable branch. Other times issues pop up in the Stable version of the language that weren't in Beta, keeping us toolchain developers on our toes and mindful of what/when we commit changes to the code base.
Testing and deploying the Rust toolchain involves several steps: fetching the toolchain from upstream, building it locally, modifying the Android source code to be compatible with the latest release, testing the build, and uploading compiler prebuilts to Gerrit. Since the Beta source is only used for testing and preparation, only a subset of the steps are performed and the commands used differ slightly from those used during a Stable release. We enumerate the steps below and show the distinct workflows. The Beta Workflow is when the toolchain developer is targeting the Beta version of the toolchain for the purposes of diagnosing possible issues. The Stable Workflow is when the toolchain developer is targeting the Stable version of the Rust toolchain for the purpose of providing prebuilts for other developers.
rust-toolchain branch.The Rust toolchain has two branch manifests in AOSP, rust-toolchain and master-plus-rust. We will be using the master-plus-rust branch in this documentation for two reasons: 1) testing the prebuilts to ensure they can build Android and 2) generating profiles for Profile Guided Optimization.
The following steps need to be done once when setting up. The following commands will create a repo directory (e.g. ~/master-plus-rust), initialize it with the master-plus-rust manifest, and synchronize the repository.
$ mkdir rust-toolchain $ cd rust-toolchain $ repo init -u https://android.googlesource.com/platform/manifest -b master-plus-rust $ repo sync -j16
Each time you start to work on updating the toolchain you will want to sync in the toolchain and aosp directory.
$ cd $TOOLCHAIN $ repo sync -j16
It can also be helpful to initialize the Android build environment:
$ . build/envsetup.sh $ lunch <build target; e.g. aosp_cf_x86_64_phone-userdebug>
Commands below will assume you have performed this initialization. You may select a new lunch target at any point.
It is best to start the process with a freshly-synchronized repository.
$ ./toolchain/android_rust/fetch_source.py -i <issue_number> <--beta> <rust_version> # e.g. ./toolchain/android_rust/fetch_source.py 1.57.0
The fetch_source.py script has several more useful options, including support for overwriting existing branches, manually specifying the branch name, or fetching beta/nightly archives. Details are listed in the --help output.
Note that you can check when the branches are updated on github.
$ ./toolchain/android_rust/build.py --lto thin
Below are some of the options that can be passed to the build script:
--build-name - Specify the name of the resulting build archive. E.g. passing test-build will result in the archive name rust-test-build.tar.gz--lto {none,thin,full} - If LTO should be enabled and which mode it should operate under--gc-sections - If functions/data should be split into different sections and garbage collected during linking--llvm-linkage {static,shared} - If the LLVM library should be linked statically or dynamicallyThe prebuilts used for release are compiled with the --lto thin, --gc-sections and --llvm-linkage static flags, in addition to PGO flags described below in the PGO section.
There are various reasons why the build might break. Check out Section 2.1 for helpful instructions and tips to build Rust.
If the build seems to be going, this will take a while; switch to another task, get some coffee, etc.
By default, this step will create rust-dev-tar.gz.
Tips: - Use the same branch name as you did for rustc if you want repo tooling to help you later. E.g. rust-update-source-1.59.0
Note, in order to include a particular CL in your update. Go to the ‘download’ tab in the CL, copy the cherry pick script, and paste that in the relevant git repo.
The test_compiler.py script can be used to verify a compiler prebuilt. E.g.
$ ./toolchain/android_rust/test_compiler.py --prebuilt-path dist/rust-dev.tar.gz \ --target aosp_cf_x86_64_phone-userdebug --image
The --image flag will cause a full device image to be built and will result in significantly longer test times. It is only necessary to pass this flag if you intend to do a boot test.
There are various things that can break when testing a new prebuilt. Take a look at Section 2.3 for instructions on how to fix the build.
If --image was passed to the test_compiler.py script during the previous step you can now use the generated image to boot a device. To boot a cuttlefish virtual device you would use the following command:
acloud create --local-image
Once it is complete, this command will print the IP address and port that can be used to connect to the virtual device. Export this is as ANDROID_SERIAL or use it as a command line argument to Android development tools.
Use the --local-instance flag to run the virtual device on the machine running the command. This is not recommended for Google developers (see).
Repeat steps Test-1 and Test-2 to fetch the Rust source you wish to release. This will usually not be a beta release so ensure that the -b flag isn‘t passed to the fetch_source.py script unless that’s really what you intended. You might also need to use the --overwrite flag if you've worked on this release before, say during the Testing Workflow
Place any changes to toolchain/android_rust and toolchain/rustc into a topic, and use repo to upload as you usually would:
repo upload -t -o l=Presubmit-Ready+1 # You may need to use `-o nokeycheck` too for cargo prebuilts.
This may take a while because updates to rustc can be hefty in size. Double check the response from the server to make sure the change went through.
You‘ll need to get these changes +2’d and merged before you can proceed.
Make sure the same topic has been added to both CLs with an updated Rust version number, e.g. rust-update-source-1.59.0.
Note the Commit message should include the Test strategy and buganizer ID, such as
Test: ./build.py --lto thin Bug: 232437287
Wait for the PGO pipeline to complete a green build including your changes. Next, wait for the next rust toolchain build after the PGO pipeline build including the new source completed. It will often be necessary to submit an empty CL to trigger a build on the rust-toolchain branch.
$ git commit --allow-empty
The next step (Release-8) can be done concurrently.
Follow steps Test-3, and Test-4 in the Testing workflow.
Use the following command to fetch, unpack, and commit the prebuilts and updated references to the prebuilts:
$ ./toolchain/android_rust/update_prebuilts.py --chained -i <issue_number> <rust_version>
By default the script will fetch artifacts from the last known good build. If you wish to specify a specific build you can use the --bid <bid> flag. The --chained flag tells the script to fetch artifacts from the build at the end of the PGO pipeline instead of the non-PGO version.
Upload the new commits in prebuilts/rust, build/soong, toolchain/android_rust, and any of the manually updated locations to Gerrit. Do this by going into both directories and using:
$ repo upload --cbr .
Note that as a last argument cbr will upload the current branch and . will upload this directory. Note that the terminal might flag an issue at this step with f.. instructions, but you can just follow the instructions from the command line and retry this step.
This process should create three CLs. Open up the Gerrit links and mark one of them as presubmit-ready. Add the same topic to both CLs with an updated Rust version number, e.g. rust-update-prebuilts-1.59.0.
If the new release is a point or patch release (e.g. 1.60.1/1.60.1.p1) you may skip this step.
If this is a new minor release (e.g. 1.61.0) then start by inspecting the preceding release. If it contains any point or patch release delete all but the last one. This can be done either in one large CL or in individual CLs, though the latter is preferred.
A release can be removed using the following commands:
$ cd prebuilts/rust $ repo start rust-remove-$OLD_RUST_VERSION $ git rm -rf {linux-x86,linux-musl-x86,darwin-x86}/$OLD_RUST_VERSION $ git commit $ repo upload .
Commit messages are usually formatted as follows:
Removing Rust $OLD_RUST_VERSION prebuilts Bug: <bug ID> Test: m rust
Prebuilts before version 1.61.0.p1 will not include linux-musl-x86 binaries so it will be necessary to remove that target from the above command until 1.60.0 is removed.
After the preceding release is cleaned up remove older releases until only the new release and the preceding release remain.
Once the CL containing the new prebuilts has been merged it needs to be tagged. This tag is not used by Android, but Chrome is using it to produce an MPM of our compiler releases for their work.
These commands do not have a review step like uploading a change, so be sure that you have landed (not just uploaded) the commits from the previous step.
$ cd prebuilts/rust $ git tag rustc-$RUST_VERSION $ git push aosp rustc-$RUST_VERSION
The new compiler will now be automatically made available to Chrome. (Actually rolling the version of the compiler they‘re using is up to them, you don’t need to worry about that part.)
You can check for tags here.
While updating the Rust toolchain there are various issues that can arise. Things can break, roadblocks can get in the way, and others might need to be brought in. In this section we describe the different types of issues that can occur, examples, and instruct on how to move past them.
They are organized by topics: - Section 2.1: Itemizes various breaks that can occur when building Rust during Step Test-2 Build locally. - Section 2.2: Describes roadblocks hit when testing during Step Test-3:Test, which is a part of Step Release-8: Testing Build - Section 2.3: Proves some notes on updating the prebuilts during Step Release-9:Update Prebuilts - Section 2.4: Describes issues with uploading to Gerrit as done in both Step Release 6:Upload and Step Release 10:Upload to Gerrit
Things that can break while building the Rust toolchain:
Patch Application
The Android project carries several patches for the Rust toolchain source. Patches make changes to a code file. In order for a patch file to be successfully applied the code that it is targeting needs to match the code file closely enough for the algorithm in the patch program to identify the relevant code. If a patch file fails to be applied then it is likely due to a change in the targeted code base. In which case, the next step is to figure out if the patch is still necessary to be applied or if that Patch file can be removed.
To know which patch file failed take a look at the terminal error message. The error message will also say what hunk # and name of the Rust file. Open up the patch file in android_rust/patches and the Rust file. At this point it'll either look like (1) patch was already applied, (2) the codebase was otherwise modified.
In the case of (1), it would be useful to verify that the patch was applied. This can be done by look at the log history for that Rust file.If you do believe that the patch was merged upstream then you just need to remove the patch from the patches/ directory, e.g.
pushd toolchain/android_rust repo start update-rustc-$RUST_VERSION git rm patches/rustc-000n-Already-merged.patch git commit -m "Remove Foo patch that has landed upstream" popd
In the case of (2), the codebase was changed in some way. Sometimes the difference might be very simple. For instance, one time the difference was just a variable name change and looking at the log history confirmed that.
To be able to successfully apply the patch the code in the patch must match the code base exactly so go ahead and modify the patch code to match the code base. If adding or removing lines of code be sure to update the number of lines that is noted in the patch file. If you are unsure if the change upholds the intent of the patch go ahead and email the patch owner, but note they will also be added as a reviewer.
After editing the patch file, upload the change to Gerrit, and ask the patch owner to review the changes to make sure the intent of the patch is still Upheld. Use the topic with “source”.
Here is an example of updating a patch file to correspond to changes in code CL.
It may also be useful to create a new patch. The following:
git format-patch HEAD~
will generate a patch file for just the previous commit.
Directory Structure Change Sometimes another library needs to be imported. We can do this by adding to the STDLIB_SOURCES definition in the do_build.py script.
For example we got the following:
error: couldn't read prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/../../portable-simd/crates/core_simd/src/mod.rs: No such file or direct ory (os error 2) --> prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/lib.rs:415:1 | 415 | mod core_simd; | ^^^^^^^^^^^^^^ error: aborting due to previous error 22:37:34 ninja failed with: exit status 1 #### failed to build some targets (18 seconds) ####
and as a result we added the portable-simd library as seen in this CL.
Binary Incompatibility
TODO: text here to describe how the user will know this type of error occurred and how to fix it
Compilation Failure
TODO: text here to describe how the user will know this type of error occurred and how to fix it
New Crate
Sometimes a new crate is added and a modification needs to be made to prebuilts/rustc/Android.bp.
TODO: text here to describe how the user will know this type of error occurred and how to fix it
New Flags
The solution might be to disable a new flag that have been added to the build process. Example CL
Check the out file CMakeCache.txt to see any additional flags that may have been added during the build process. Try testing the new flags with the older version of the compiler.
Things that can break during testing:
Hermeticity Breakage
A Hermeticity breakage occurs when the build uses a tool or header that is different from what we expect. For instance, a crate including a build script allowing a different compiler/library to be used than expected. Some of these issues arise because of bugs in the upstream build system. Others can also be caused because the environment they are built-in have certain defaults set and carry unintended expectations.
TODO @Chriswailes: Describle how to approach this type of problem, given lzma-sys example.
Build System Breakage
There can be build breakage issues.
For instance, needing to change "-C passes='sancov'", to "-C passes='sancov-module'", such as in this CL.
Android Source Warnings
A common issue is that this step triggers new warnings on existing source files. If the compiler suggests a fix, apply it. Otherwise make the most reasonable looking change necessary to keep the compiler happy and rely on the original author to determine correctness during code review.
Here is an example of the workflow to modify file x.rs:
# navigate to the repo with file x.rs repo start rust-1.59.0-fixes # make changes to x.rs git add x.rs git commit
The commit message should include the testing strategy and buganizer ticket number. Here is an example commit message from one of these types of CLs.
Changes for the Rust 1.59.0 update bug: 215232614 Test: TreeHugger and compiling with m rust Change-Id: I1d25f5550f60ff1046a3a312f7bd210483bf9364
Note, that it is preferable to create one commit per big change to a repo, so it might be helpful to use amend when adding more changes to the code:
$ git commit --amend --no-edit
After committing the changes continue to upload them to Gerrit with
repo upload .
In Gerrit for the corresponding CL include the owners of the file as reviewers and do not set a topic.
Next we will need to periodically check-in on these CLs. If they pass presubmit, then we are waiting for the CLs to be approved/submitted by the file owners. This can take a few days and may require a friendly nudge.
If the files do not pass presubmit then the changes may not have been backwards compatible with Rust and we will need to compile it with the latest version of Rust. If that is the case on Gerrit include it in the topic rust-update-prebuilts-1.59.0, with an updated Rust version number.
We are not able to move to the final Step 12 (tagging) until these CLs created in this process has been submitted/accepted.
Clipper errors are typically pretty straightforward but sometimes it’s not clear what to change. Try checking at the change history or how the change has been implemented in other parts of the Rust codebase. For instance, builder.config.llvm_link_shared was changed to !builder.llvm_link_shared().
Miscompilation
A miscompilation may have occured when it successfully compiles but the devices fail to boot or pass CTS tests.
TODO: text here to describe how the user will know this type of error occurred and how to fix it
Deprecated flag
The following error:
warning: `-Z symbol-mangling-version` is deprecated; use `-C symbol-mangling-version`
led to changes where that variable was used with -Z was changed to -C in rust/config/global.go. CL .
Things that can break when updating the prebuilts:
File undefined
Try deleting the branch git branch -D rust-update-prebuilts-1.59.0-local
Fetching ID shell : > ./toolchain/android_rust/update_prebuilts.py --chained 1.62.0 Unable to fetch LKGB build ID This means you forgot to run gcert.
unknown manifest Note that some tests might fail during presubmit because of code that live in other manifests. Make the apprioriate changes as usual, but in the right place.
Things that can break while uploading files to Gerrit:
Geritt Limitation
Help, Gerrit won't take my update!
First, try again. Sometimes Gerrit is just flaky and will take it on the second or third try.
If that‘s still not working, you are likely hitting a size limitation (for example, because rustc updated it’s LLVM revision, so the diff is bigger than usual). In this case, you will need to work with the build team to get them to use a “direct push” to skip Gerrit's hooks. Look at the initial import bug for an example conversation about importing oversized changes.
Detached head
Use this to get away from detached head:
git branch -u aosp/master
Run test locally
If presbuilts failed presubmit, one thing you can do is try to run test locally
source build/envsetup.sh lunch aosp_cf_x86_64_phone-userdebug m acloud create --local-image export ANDROID_SERIAL=<value printed by previous command> atest bytes_test_tests_test_bytes // or whatever tests are failing in the prebuilts
Output Format
There could be an error parsing the results of running Rust tests.
A prior release Rust v.1.61 release notes include messages with ignored tests. We had to modify how the trade federation code analyzes the result of running tests (changed regex). CL The Gerrit CL triggered the same changes in Critique.
Question: Should this be added to Section 2.1 Build Error - Missing Crate or is it a different type of error?
If the sysroot build is broken, check whether the error mentions a missing crate. If it does, there have likely been new components added to the sysroot. To address this, you will need to:
STDLIB_SOURCES in toolchain/android_rust/build.py.DIST_DIR=$TOOLCHAIN/dist ./toolchain/android_rust/build.py$DIST_DIRprebuilts/rust in your Android tree and use git fetch local followed by a checkout to get it imported into your Android tree.prebuilts/rust/Android.bp. Except for publicly exported crates (which you're not adding right now), all modules in this file must be suffixed with .rust_sysroot to avoid confusion with user crates. Dependency edges should all be of rlib form unless depending on a publicly exported crate, in which case the dependency edge should match the type of the final product. As examples, libterm (exported) depends by dylib on libstd, but libterm.static (also exported) depends by rlib on libstd.static.libhashbrown.rust_sysroot is built only as an rlib, and is linked as an rlib everywhere it is used.TODO: Merge this text into the previous sections
New lints/clippys can cause build breakage and may require significant refactoring as the code base grows. To avoid blocking toolchain upgrades, explicitly allow the breaking lints/clippys when first upgrading the toolchain.
build/soong/rust/config/lints.go with the -A flag.Android‘s Rust toolchain is built using profile guided optimization (PGO). This five-stage process records execution traces from compiling Android’s Rust codebase and uses this information to improve the performance of the Rust compiler and toolchain libraries.
In the first stage the Rust toolchain is built with the profile collection instrumentation. If the toolchain is built with shared LLVM linkage the resulting compiler and libraries will produce sets of profile data in the specified output directory: one for LLVM and one for Rust. If, on the other hand, the toolchain is built with static LLVM linkage the toolchain will only produce profile data for the Rust components. This is due to the fact that the statically linked LLVM objects will need to be optimized using the profile data for the executable or library they are being linked into.
The toolchain produced in Stage 1 is then used to compile all of the Rust code in the Android codebase for multiple architectures. This will produce one or more sets of profile data that is then merged together.
The profile data from Stage 2 is then used to re-compile the Rust toolchain using PGO. In addition, the LLVM libraries are instrumented to record context-sensitive profile-guided optimization (CS-PGO) information. This stage will produce a LLVM profile if the toolchain is built with shared LLVM linkage or a Rust profile if built with static linkage.
Android's Rust code is re-built using the CS-PGO instrumented compiler and the results are merged with the profiles generated during Stage 2.
The final compiler is built using the profile produced in Stage 4.
The PGO pipeline can be executed using the pipeline.py script. No command line arguments are necessary, though the --build-name and --dist-path arguments can be used to set the prebuilt archive name and output directory respectively. This process will take six or more hours to complete, depending on hardware.
Profile data is collected from three work sets:
The PGO pipeline requires the Rust and C++ compilers to emit the same version of LLVM bitcode. This is not always the case, as Rust uses an internal copy of LLVM to generate code in rustc. It is this internal LLVM library that is compiled by Android's Clang/LLVM prebuilts. If the LLVM library and the code it emits use incompatible versions of LLVM IR or the runtime libraries then lld will be unable to link the toolchain.
Starting with Rust 1.56.1 it is possible to reproduce Android‘s Rust toolchain prebuilts. To do this you will start with the prebuilt’s manifest, which is located in the prebuilts/rust/linux-x86/<version> directory. Starting with Rust 1.61.0 the manifest is also available in the toolchain/android_rust/artifacts/<version> directory.
To begin reproducing a given build navigate to a new directory and run the following command:
$ repo init -m /path/to/manifest.xml -b rust-toolchain && repo sync -c -j16
To reproduce toolchains before Rust version 1.61.0 run the following command:
$ ./toolchain/android_rust/build.py --lto thin --llvm-linkage shared
To reproduce toolchains version 1.61.0 and above use the build commands listed in the toolchain/android_rust/artifacts/<version>/rust_build_command.<target>.<build id>.sh files. Be sure to replace the paths to the PGO profiles from the saved commands with paths to the profiles in the artifacts directory.
The Rust toolchain includes a list of the known Rust targets in Android. This is currently only used when processing Soong traces for performance information and many of the targets can be discovered automatically so an out of date file is not a major issue. To increase the accuracy of any parsed traces you can update the module list using the following commands:
$ m queryview
$ bazel query --config=queryview 'kind("rust_(?!default).*", //...)' --output label_kind > toolchain/android_rust/android-rust-modules.txt