| # The Android Platform Build System: Structure and Design |
| |
| This document consolidates information about the Android Platform Build System, its evolution, key components, and the specific process for converting products to a "Soong-only" build. This comprehensive overview is intended as input for a Gemini-CLI agent to understand the system and assist in verifying if a specific Android product (represented as `<PRODUCT_NAME>`) can be successfully converted to a "Soong-only" build. |
| |
| ## 1. Introduction to the Platform Build System |
| |
| The Android Platform Build System is responsible for building the system images for Android devices. It can also build "unbundled" apps, various tests, SDKs, and miscellaneous tools. It is the system developers interact with locally when running commands like `lunch`, `m`, `mm`, `mma`, `mmm`, `mmma`, and editing files like `Android.mk`, `Android.bp`, and `AndroidProducts.mk`. |
| |
| **What the Platform Build System is NOT**: |
| * A third-party app builder (like NDK Build, Cmake, Gradle). |
| * A Kernel, bootloader, or firmware builder, as these often have their own build systems. |
| * A system for build fleets, scheduling builds, or starting builds on a builder (like Busytown, Buildbot, Treehugger). However, it does share tools with these teams. |
| |
| The core objective of the build tooling is to transform the contents of source control into a set of filesystem images (`.img` files), the Android SDK and NDK, and other various outputs. |
| |
| ## 2. History and Evolution of the Android Build System |
| |
| Android's build system has evolved significantly over time, moving from a monolithic Make-based system to a more modular and efficient architecture incorporating Kati, Ninja, and Soong. |
| |
| ### 2.1. Android M: Monolithic Make Invocation |
| Historically, Android spent many years as a **single monolithic Make invocation**. |
| * Make is essentially a programming language. |
| * Starting every build took on the order of minutes. |
| * `mm` loads only some Makefiles, while `mma` loads all but only builds local files. |
| * `Android.mk` files have always required module names to be unique within a module type (e.g., two shared libraries could not have the same name, but a static and shared library could). |
| |
| ### 2.2. Android N: Introducing Kati and Ninja |
| The build system started replacing Make with **Kati** and **Ninja**. |
| * **Kati**: Built by the Chrome team to build Android faster. It's a "full" Make implementation that can write out the build graph instead of running it, allowing detection of when inputs haven't changed to reuse the last build graph. It extends the Make language with features like read-only variables, deprecated/obsolete variables, multiple output support, and restart support. Kati processes Makefiles and emits Ninja files. |
| * **Ninja**: An open-source build graph executor created for Chrome and now used in many places. It is responsible for the "execution phase" of Android's build, running commands generated by Soong and Kati in the correct order, and only when an input has changed. Ninja's data model is a directed, bipartite graph of build statements and their input/output files, described in fast-parsing `.ninja` files. Android's Ninja version has specific features like "validation nodes" and different symlink handling. |
| |
| ### 2.3. Android O: Integrating Soong |
| **Soong** was introduced as a replacement for the error-prone `Android.mk`, offering a syntax more similar to Bazel. |
| * Soong is **written in Go** and **built on top of Blueprint**. Blueprint is an open-source library for building build systems like Soong. |
| * Soong is much more flexible than the Make build system implementation. |
| * It has access to the module dependency graph, allowing it to make fewer assumptions and rely on real data, unlike Make which struggles with transitive operations. |
| * Soong reads device-specific configuration from a file produced by Make and writes out files for Kati's use, including global configuration and module information. |
| * For Android O, Make acted as a wrapper, calling `soong_ui`, which then orchestrated calls to Make (config), Soong (Android.bp), Kati (Android.mk), and finally Ninja. |
| |
| ### 2.4. Android P: Removing More Make Pieces |
| In Android P, further pieces of Make were removed, streamlining the build process where `soong_ui` directly orchestrated calls to Kati (config), Soong (Android.bp), Kati (CleanSpec), Kati (Android.mk), and Ninja. |
| |
| ### 2.5. Android Q: Adding 'dist' Handling |
| Android Q added another Kati execution to handle the `dist` target, which prepares a directory containing build outputs for release and distribution. |
| |
| ### 2.6. Current State |
| * Most things are now supported in Soong. |
| * Much of the native code in the core platform has switched to Soong. |
| * Java conversion started, but JNI libraries are still missing. |
| * Vendors largely remain in `Android.mk`, though Soong is considered feature-complete for their use cases. |
| * There's a long tail of binaries/tests, with a focus on libraries used by others. |
| * Code search indicates ~3k `Android.bp` files and ~23k `Android.mk` files internally, and 600 `Android.bp` and 3k `Android.mk` in AOSP. |
| * Graphs show the conversion progress for C/C++ and Java in AOSP, with Soong module counts steadily rising and Make module counts decreasing. |
| |
| ## 3. Key Components and Their Roles |
| |
| The build of Android involves several interconnected steps orchestrated by various tools. |
| |
| ### 3.1. Lunch (and similar tools like Tapas, Banchan) |
| * Used to **select how Android is built**. |
| * Implemented as a shell script in `build/envsetup.sh`. |
| * If a product is not specified, it offers a list of choices from the `COMMON_LUNCH_CHOICES` product variable, determined by `soong_ui` scanning `AndroidProducts.mk` files. |
| * Sets variables like `TARGET_PRODUCT`, `TARGET_BUILD_VARIANT`, and `TARGET_BUILD_TYPE` based on the user's choice. |
| |
| ### 3.2. soong_ui |
| * The **orchestrator of the entire build**; its job is to understand user requests and call other components in the correct order. |
| * All functions like `m` are thin wrappers around `soong_ui`. |
| * It's a complex, potentially rebuilt-on-every-build component, requiring incremental rebuilds. |
| * Involves an elaborate **bootstrapping process** due to `soong_build` itself being described in Blueprint files: |
| 1. `soong_ui.bash` builds Microfactory (a Go tool for incremental Go builds). |
| 2. Microfactory builds `soong_ui`. |
| 3. `soong_ui` scans the source tree for Blueprint and Makefiles to find products and Blueprint files to process. |
| 4. `soong_ui` creates a Ninja file to build `soong_build` and runs its tests, also describing how to call `soong_build` to emit outputs like the main Ninja files, documentation, JSON module graph, module dumps, and `BUILD.bazel` files. |
| * **Sandboxing**: `soong_ui` can run various build steps in a sandbox using `nsjail` to limit actions (e.g., disallowing writing outside the output directory, network access). Sandboxed steps include Kati, main Ninja invocation, and `soong_build`. |
| * **Globbing**: `soong_ui` works with `soong_build` to handle globbing results incrementally, using Ninja's "restat" feature to avoid re-running `soong_build` unnecessarily when only glob results, but not their content, change. |
| * **Special-cased targets**: Some targets are handled directly by `soong_ui` because their actions cannot be expressed solely in Ninja files, such as `checkbuild`, `bp2build`, `queryview`, `soong_docs`, `json-module-graph`, and `dist`. An empty list of goals defaults to the `droid` target. |
| |
| ### 3.3. Product Configuration |
| * Android builds for many devices and build variants, each requiring slightly different images. |
| * Product variables are computed after `lunch` is run, with `build/make/core/config.mk` as the entry point. |
| * **Invoked twice during the build**: |
| 1. First invocation from `dumpvars.go`: Variables are emitted to standard output (parsed by `soong_ui` for Ninja shell environment) and written to `soong.variables` JSON file (parsed by `soong_build` for `soong_config_*` module types). |
| 2. Second invocation (the "main" Kati invocation): Every Make variable set here is accessible. |
| * The product configuration system is being rewritten to use **Starlark** (Bazel's config language) instead of Make for less flexibility and fewer mistakes. `mk2rbc` converts Makefiles to Starlark (`.rbc`) files, and `rbcrun` evaluates them. |
| |
| ### 3.4. soong_build |
| * The **primary tool for generating most build commands**. |
| * Inputs: Blueprint files (`Android.bp`). |
| * Main output: A Ninja file describing the complete build. |
| * Consists of two parts: **Blueprint** (parsing, mutators, singletons, action generation) in `build/blueprint`, and **Android-specific bits** in `build/soong`. |
| * **Execution steps**: |
| 1. **Parse Blueprint files**: Creates modules as nodes without dependencies. |
| 2. **Execute Load hooks**: Modify modules, create new ones, register new module types. |
| 3. **Execute Mutators**: Mutate module data structures, create new variants, add dependency edges. Mutators can be top-down (before dependencies) or bottom-up (after dependencies). Key mutators include "arch", "os", and "deps". |
| 4. **Create Commands**: Modules and singletons create commands written to the output Ninja file. |
| * **Data flow**: `soong_build` is sandboxed, working directory set to root, and sensitive environment variables are managed to ensure reproducibility and dependency tracking. |
| * **Configuration and Context**: A "global object" (Config) contains immutable data (source tree, output directory, device, CPU architecture, product variables, shell environment) accessible throughout `soong_build`. Blueprint `Context` contains internal bookkeeping data. |
| * **Blueprint files (`Android.bp`)**: Text files with "modules" using a dictionary syntax. |
| * Each module has a **module type** (e.g., `genrule`, `cc_library`, `java_library`) and a list of **properties** (e.g., `name`, `srcs`, `out`). |
| * Properties can depend on how the module is built (e.g., `arch` for CPU architecture, `os` for operating system). |
| * Soong parses *every* Blueprint file globally, meaning a syntax error or a non-existent dependency in any file breaks the whole build (unless `ALLOW_MISSING_DEPENDENCIES` is set). |
| * Module factory calls `AddProperties()` to define settable properties. |
| * **Variants and Variations**: Modules can have multiple variants for different build "ways" (e.g., CPU architecture, OS, sanitizers, API versions, partition). A variant is a string-to-string map of variation tags and their values. Dependencies automatically redirect to corresponding variants of the dependent module unless `CreateLocalVariations()` is used. |
| * **Defaults modules**: Allow defining common properties for a group of similar modules (e.g., `cc_defaults`). These are implemented in `defaultsMutator`. |
| * **Product Variable Support**: Allows setting module properties based on product variables, but only for an allowlist of variables and properties. Uses `product_variables` property in Blueprint files (e.g., `platform_sdk_version` affecting `asflags`). |
| * **Vendor Variable Support**: A more flexible but cumbersome mechanism for product configuration to affect module properties. Involves setting Make variables (`SOONG_CONFIG_NAMESPACES`, `SOONG_CONFIG_${NAMESPACE}`, `SOONG_CONFIG_${NAMESPACE}_${NAME}`). These are then plumbed to `soong_build` in `soong.variables` under `VendorVars`. Requires defining bespoke module types (`soong_config_module_type`, `soong_config_string_variable`) to access these variables. |
| * **Generating actions and Singletons**: After mutators, `GenerateAndroidBuildActions()` is called for modules to create commands using `RuleBuilder`. Singletons have a global view of the module graph and are used for build actions that need this scope. |
| |
| ### 3.5. Kati |
| * An open-source tool that processes Makefiles and emits Ninja files. |
| * Used to build Android after GNU Make hit its limitations. |
| * **Kati modules**: Defined by setting Make variables and including a Makefile that implements the module type (e.g., `BUILD_STATIC_JAVA_LIBRARY`). |
| * **Dependency rule**: Kati modules are allowed to depend on Soong modules, but not vice-versa, enforcing a bottom-up migration to Soong. |
| * **Glue with Soong**: `soong_build` writes a Makefile (`out/soong/Android-$PRODUCT.mk`) that describes Soong modules as "prebuilt" modules for Kati to consume. It also writes `out/soong/make_vars-${TARGET_PRODUCT}.mk` containing Make variables Soong believes should be set. |
| |
| ### 3.6. Ninja |
| * A separate project responsible for the "execution phase" of Android's build, executing commands generated by Soong and Kati efficiently. |
| * Also used to build and run `soong_build` itself and generate the main Ninja file for the operating system. |
| * **Data Model**: Directed, bipartite graph of build statements (actions) and their input/output files, described in `.ninja` files. Uses "rules" as templates for build statements. |
| * Supports parsing `.d` files from C++ compilers to determine actual dependencies. |
| * **Execution**: Traverses the graph, checks input changes (`stat()`), and re-runs build statements if inputs change. |
| * **`restat` mechanism**: Allows not re-running downstream build statements if an output file's content (not just mtime) hasn't changed. This is different from Bazel, which uses checksums. |
| |
| ### 3.7. Bazel |
| * The **newest build system** being integrated. |
| * **`bp2build`**: Converts Blueprint files (`Android.bp`) to `BUILD.bazel` files, mapping Soong modules to Bazel targets. |
| * Current rule: Soong modules can depend on Bazel targets, but not the other way around, similar to the Make-Soong migration strategy. |
| * **Mixed builds**: It's possible to run a build where converted modules are processed by Bazel and the rest by Soong, generating Ninja actions for both. `soong_build` uses "bazel cquery" and "bazel aquery" to get information from Bazel about transitive closures and actions. |
| |
| ### 3.8. Packaging |
| * The "packaging" Kati call prepares the `distdir` directory, which contains the "official" build outputs for release and distribution. |
| * Main input: `out/target/product/$PRODUCT/obj/CONFIG/kati_packaging/dist.mk`. |
| * Produces `out/build-$PRODUCT-package.ninja` containing `cp` invocations to assemble the `distdir`. |
| * The `dist.mk` file is written during the main Kati invocation by `soong_build` based on `AndroidMkEntries.GetDistForGoals()`. |
| |
| ### 3.9. Cleanspec |
| * A mechanism that appears to remove stale output files, though its exact workings are not fully understood by all developers. |
| |
| ## 4. Namespacing Modules in Soong |
| |
| Namespacing in Soong addresses the problem of module name conflicts, especially when mixing code from different vendors or different revisions of the same vendor's code. |
| |
| ### 4.1. Problem Background |
| * Historically, `Android.mk` required module names to be unique within a module type. |
| * Soong and Blueprint extended this requirement: module names must be unique across *all* module types. |
| * This uniqueness is reasonable for a single device build, as the module name is tied to a unique install location. |
| * However, it becomes problematic when integrating multiple devices into the same source tree, as similar SoCs may have different vendor code versions with identical module names due to varying release schedules. |
| * Prior workarounds (defining conflicting modules only for the currently-building device) led to difficulties in build confidence and more breakages. |
| |
| ### 4.2. Namespacing Objective |
| * Put modules from different vendors (or different revisions of the same vendor's code) into namespaces so that they don’t conflict but can still be built in parallel. |
| |
| ### 4.3. How Namespacing Works |
| * Responsibility for mapping names to modules moves from Blueprint to Soong. |
| * **`soong_namespace` module**: A namespace is declared at any point in an `Android.bp` file where a `soong_namespace` module is present. It affects all modules in that `Android.bp` file and its subdirectories. Modules cannot be declared before the namespace module. |
| * **`imports` property**: A `soong_namespace` can declare an `imports` property for other namespaces. **Imported namespaces are not transitive**, and circular imports are possible. |
| * **Independence**: Namespaces are not inherited or embedded; child namespaces are independent, requiring an import or fully qualified name for references. |
| * **Implicit root namespace**: Contains any modules defined outside all explicit namespaces and is implicitly imported by all other namespaces. |
| * **Name Lookups**: |
| * **Short module name**: Used for modules within the same namespace or in an imported namespace. |
| * **Fully-qualified name**: Used for modules outside the current namespace, in the form `//path/to/module:module_name`. |
| * `ModuleFromName` lookups starting with `//` use the fully-qualified name map; otherwise, `nameData` is used to find current and imported namespaces, checking the short name map. |
| * **Core platform code**: All core platform code will reside in a single root namespace and can continue to use short names. |
| * **Implicit module dependencies**: Always need to be fully-qualified names to ensure accessibility from any namespace. |
| * **Export to Android.mk**: Module names must be unique when exported to `Android.mk`. Products select which Soong namespaces they need, and only modules from **selected namespaces** will be exported. Modules in unselected namespaces are still built as part of `make checkbuild`. |
| * **Implementation Details**: |
| * Blueprint adds a `NameInterface` allowing Soong to map names to `blueprint.Module` instances. |
| * Soong's `NameInterface` implementation stores maps for module lookup. |
| * Each module gets a "fully-qualified name" (`//` + `ctx.ModuleDir()` + `:` + `ModuleName()`). |
| * Each module also gets a "short name" stored as a `(namespace, name)` tuple. Both fully-qualified and short names are used as keys in a map to the `blueprint.Module`. |
| * When a `Namespace` type module is encountered, Soong uses `ctx.SetNameData()` to store the new namespace, affecting subsequent modules. |
| |
| ### 4.4. Future Improvements and Caveats |
| * **Visibility rules**: Could be implemented using a `soong_visibility` module, enforced at name lookup or a later check stage. |
| * **Name conflicts**: Semantics for name conflicts in imported namespaces need to be defined, likely requiring uniqueness across imported namespaces for `Android.mk` export. |