The Android Platform Build System: Structure and Design

This document consolidates information about the Android Platform Build System, its evolution, key components, and the specific process for converting products to a “Soong-only” build. This comprehensive overview is intended as input for a Gemini-CLI agent to understand the system and assist in verifying if a specific Android product (represented as <PRODUCT_NAME>) can be successfully converted to a “Soong-only” build.

1. Introduction to the Platform Build System

The Android Platform Build System is responsible for building the system images for Android devices. It can also build “unbundled” apps, various tests, SDKs, and miscellaneous tools. It is the system developers interact with locally when running commands like lunch, m, mm, mma, mmm, mmma, and editing files like Android.mk, Android.bp, and AndroidProducts.mk.

What the Platform Build System is NOT:

  • A third-party app builder (like NDK Build, Cmake, Gradle).
  • A Kernel, bootloader, or firmware builder, as these often have their own build systems.
  • A system for build fleets, scheduling builds, or starting builds on a builder (like Busytown, Buildbot, Treehugger). However, it does share tools with these teams.

The core objective of the build tooling is to transform the contents of source control into a set of filesystem images (.img files), the Android SDK and NDK, and other various outputs.

2. History and Evolution of the Android Build System

Android's build system has evolved significantly over time, moving from a monolithic Make-based system to a more modular and efficient architecture incorporating Kati, Ninja, and Soong.

2.1. Android M: Monolithic Make Invocation

Historically, Android spent many years as a single monolithic Make invocation.

  • Make is essentially a programming language.
  • Starting every build took on the order of minutes.
  • mm loads only some Makefiles, while mma loads all but only builds local files.
  • Android.mk files have always required module names to be unique within a module type (e.g., two shared libraries could not have the same name, but a static and shared library could).

2.2. Android N: Introducing Kati and Ninja

The build system started replacing Make with Kati and Ninja.

  • Kati: Built by the Chrome team to build Android faster. It‘s a “full” Make implementation that can write out the build graph instead of running it, allowing detection of when inputs haven’t changed to reuse the last build graph. It extends the Make language with features like read-only variables, deprecated/obsolete variables, multiple output support, and restart support. Kati processes Makefiles and emits Ninja files.
  • Ninja: An open-source build graph executor created for Chrome and now used in many places. It is responsible for the “execution phase” of Android‘s build, running commands generated by Soong and Kati in the correct order, and only when an input has changed. Ninja’s data model is a directed, bipartite graph of build statements and their input/output files, described in fast-parsing .ninja files. Android's Ninja version has specific features like “validation nodes” and different symlink handling.

2.3. Android O: Integrating Soong

Soong was introduced as a replacement for the error-prone Android.mk, offering a syntax more similar to Bazel.

  • Soong is written in Go and built on top of Blueprint. Blueprint is an open-source library for building build systems like Soong.
  • Soong is much more flexible than the Make build system implementation.
  • It has access to the module dependency graph, allowing it to make fewer assumptions and rely on real data, unlike Make which struggles with transitive operations.
  • Soong reads device-specific configuration from a file produced by Make and writes out files for Kati's use, including global configuration and module information.
  • For Android O, Make acted as a wrapper, calling soong_ui, which then orchestrated calls to Make (config), Soong (Android.bp), Kati (Android.mk), and finally Ninja.

2.4. Android P: Removing More Make Pieces

In Android P, further pieces of Make were removed, streamlining the build process where soong_ui directly orchestrated calls to Kati (config), Soong (Android.bp), Kati (CleanSpec), Kati (Android.mk), and Ninja.

2.5. Android Q: Adding ‘dist’ Handling

Android Q added another Kati execution to handle the dist target, which prepares a directory containing build outputs for release and distribution.

2.6. Current State

  • Most things are now supported in Soong.
  • Much of the native code in the core platform has switched to Soong.
  • Java conversion started, but JNI libraries are still missing.
  • Vendors largely remain in Android.mk, though Soong is considered feature-complete for their use cases.
  • There's a long tail of binaries/tests, with a focus on libraries used by others.
  • Code search indicates ~3k Android.bp files and ~23k Android.mk files internally, and 600 Android.bp and 3k Android.mk in AOSP.
  • Graphs show the conversion progress for C/C++ and Java in AOSP, with Soong module counts steadily rising and Make module counts decreasing.

3. Key Components and Their Roles

The build of Android involves several interconnected steps orchestrated by various tools.

3.1. Lunch (and similar tools like Tapas, Banchan)

  • Used to select how Android is built.
  • Implemented as a shell script in build/envsetup.sh.
  • If a product is not specified, it offers a list of choices from the COMMON_LUNCH_CHOICES product variable, determined by soong_ui scanning AndroidProducts.mk files.
  • Sets variables like TARGET_PRODUCT, TARGET_BUILD_VARIANT, and TARGET_BUILD_TYPE based on the user's choice.

3.2. soong_ui

  • The orchestrator of the entire build; its job is to understand user requests and call other components in the correct order.
  • All functions like m are thin wrappers around soong_ui.
  • It's a complex, potentially rebuilt-on-every-build component, requiring incremental rebuilds.
  • Involves an elaborate bootstrapping process due to soong_build itself being described in Blueprint files:
    1. soong_ui.bash builds Microfactory (a Go tool for incremental Go builds).
    2. Microfactory builds soong_ui.
    3. soong_ui scans the source tree for Blueprint and Makefiles to find products and Blueprint files to process.
    4. soong_ui creates a Ninja file to build soong_build and runs its tests, also describing how to call soong_build to emit outputs like the main Ninja files, documentation, JSON module graph, module dumps, and BUILD.bazel files.
  • Sandboxing: soong_ui can run various build steps in a sandbox using nsjail to limit actions (e.g., disallowing writing outside the output directory, network access). Sandboxed steps include Kati, main Ninja invocation, and soong_build.
  • Globbing: soong_ui works with soong_build to handle globbing results incrementally, using Ninja's “restat” feature to avoid re-running soong_build unnecessarily when only glob results, but not their content, change.
  • Special-cased targets: Some targets are handled directly by soong_ui because their actions cannot be expressed solely in Ninja files, such as checkbuild, bp2build, queryview, soong_docs, json-module-graph, and dist. An empty list of goals defaults to the droid target.

3.3. Product Configuration

  • Android builds for many devices and build variants, each requiring slightly different images.
  • Product variables are computed after lunch is run, with build/make/core/config.mk as the entry point.
  • Invoked twice during the build:
    1. First invocation from dumpvars.go: Variables are emitted to standard output (parsed by soong_ui for Ninja shell environment) and written to soong.variables JSON file (parsed by soong_build for soong_config_* module types).
    2. Second invocation (the “main” Kati invocation): Every Make variable set here is accessible.
  • The product configuration system is being rewritten to use Starlark (Bazel's config language) instead of Make for less flexibility and fewer mistakes. mk2rbc converts Makefiles to Starlark (.rbc) files, and rbcrun evaluates them.

3.4. soong_build

  • The primary tool for generating most build commands.
  • Inputs: Blueprint files (Android.bp).
  • Main output: A Ninja file describing the complete build.
  • Consists of two parts: Blueprint (parsing, mutators, singletons, action generation) in build/blueprint, and Android-specific bits in build/soong.
  • Execution steps:
    1. Parse Blueprint files: Creates modules as nodes without dependencies.
    2. Execute Load hooks: Modify modules, create new ones, register new module types.
    3. Execute Mutators: Mutate module data structures, create new variants, add dependency edges. Mutators can be top-down (before dependencies) or bottom-up (after dependencies). Key mutators include “arch”, “os”, and “deps”.
    4. Create Commands: Modules and singletons create commands written to the output Ninja file.
  • Data flow: soong_build is sandboxed, working directory set to root, and sensitive environment variables are managed to ensure reproducibility and dependency tracking.
  • Configuration and Context: A “global object” (Config) contains immutable data (source tree, output directory, device, CPU architecture, product variables, shell environment) accessible throughout soong_build. Blueprint Context contains internal bookkeeping data.
  • Blueprint files (Android.bp): Text files with “modules” using a dictionary syntax.
    • Each module has a module type (e.g., genrule, cc_library, java_library) and a list of properties (e.g., name, srcs, out).
    • Properties can depend on how the module is built (e.g., arch for CPU architecture, os for operating system).
    • Soong parses every Blueprint file globally, meaning a syntax error or a non-existent dependency in any file breaks the whole build (unless ALLOW_MISSING_DEPENDENCIES is set).
    • Module factory calls AddProperties() to define settable properties.
  • Variants and Variations: Modules can have multiple variants for different build “ways” (e.g., CPU architecture, OS, sanitizers, API versions, partition). A variant is a string-to-string map of variation tags and their values. Dependencies automatically redirect to corresponding variants of the dependent module unless CreateLocalVariations() is used.
  • Defaults modules: Allow defining common properties for a group of similar modules (e.g., cc_defaults). These are implemented in defaultsMutator.
  • Product Variable Support: Allows setting module properties based on product variables, but only for an allowlist of variables and properties. Uses product_variables property in Blueprint files (e.g., platform_sdk_version affecting asflags).
  • Vendor Variable Support: A more flexible but cumbersome mechanism for product configuration to affect module properties. Involves setting Make variables (SOONG_CONFIG_NAMESPACES, SOONG_CONFIG_${NAMESPACE}, SOONG_CONFIG_${NAMESPACE}_${NAME}). These are then plumbed to soong_build in soong.variables under VendorVars. Requires defining bespoke module types (soong_config_module_type, soong_config_string_variable) to access these variables.
  • Generating actions and Singletons: After mutators, GenerateAndroidBuildActions() is called for modules to create commands using RuleBuilder. Singletons have a global view of the module graph and are used for build actions that need this scope.

3.5. Kati

  • An open-source tool that processes Makefiles and emits Ninja files.
  • Used to build Android after GNU Make hit its limitations.
  • Kati modules: Defined by setting Make variables and including a Makefile that implements the module type (e.g., BUILD_STATIC_JAVA_LIBRARY).
  • Dependency rule: Kati modules are allowed to depend on Soong modules, but not vice-versa, enforcing a bottom-up migration to Soong.
  • Glue with Soong: soong_build writes a Makefile (out/soong/Android-$PRODUCT.mk) that describes Soong modules as “prebuilt” modules for Kati to consume. It also writes out/soong/make_vars-${TARGET_PRODUCT}.mk containing Make variables Soong believes should be set.

3.6. Ninja

  • A separate project responsible for the “execution phase” of Android's build, executing commands generated by Soong and Kati efficiently.
  • Also used to build and run soong_build itself and generate the main Ninja file for the operating system.
  • Data Model: Directed, bipartite graph of build statements (actions) and their input/output files, described in .ninja files. Uses “rules” as templates for build statements.
  • Supports parsing .d files from C++ compilers to determine actual dependencies.
  • Execution: Traverses the graph, checks input changes (stat()), and re-runs build statements if inputs change.
  • restat mechanism: Allows not re-running downstream build statements if an output file‘s content (not just mtime) hasn’t changed. This is different from Bazel, which uses checksums.

3.7. Bazel

  • The newest build system being integrated.
  • bp2build: Converts Blueprint files (Android.bp) to BUILD.bazel files, mapping Soong modules to Bazel targets.
  • Current rule: Soong modules can depend on Bazel targets, but not the other way around, similar to the Make-Soong migration strategy.
  • Mixed builds: It's possible to run a build where converted modules are processed by Bazel and the rest by Soong, generating Ninja actions for both. soong_build uses “bazel cquery” and “bazel aquery” to get information from Bazel about transitive closures and actions.

3.8. Packaging

  • The “packaging” Kati call prepares the distdir directory, which contains the “official” build outputs for release and distribution.
  • Main input: out/target/product/$PRODUCT/obj/CONFIG/kati_packaging/dist.mk.
  • Produces out/build-$PRODUCT-package.ninja containing cp invocations to assemble the distdir.
  • The dist.mk file is written during the main Kati invocation by soong_build based on AndroidMkEntries.GetDistForGoals().

3.9. Cleanspec

  • A mechanism that appears to remove stale output files, though its exact workings are not fully understood by all developers.

4. Namespacing Modules in Soong

Namespacing in Soong addresses the problem of module name conflicts, especially when mixing code from different vendors or different revisions of the same vendor's code.

4.1. Problem Background

  • Historically, Android.mk required module names to be unique within a module type.
  • Soong and Blueprint extended this requirement: module names must be unique across all module types.
  • This uniqueness is reasonable for a single device build, as the module name is tied to a unique install location.
  • However, it becomes problematic when integrating multiple devices into the same source tree, as similar SoCs may have different vendor code versions with identical module names due to varying release schedules.
  • Prior workarounds (defining conflicting modules only for the currently-building device) led to difficulties in build confidence and more breakages.

4.2. Namespacing Objective

  • Put modules from different vendors (or different revisions of the same vendor's code) into namespaces so that they don’t conflict but can still be built in parallel.

4.3. How Namespacing Works

  • Responsibility for mapping names to modules moves from Blueprint to Soong.
  • soong_namespace module: A namespace is declared at any point in an Android.bp file where a soong_namespace module is present. It affects all modules in that Android.bp file and its subdirectories. Modules cannot be declared before the namespace module.
  • imports property: A soong_namespace can declare an imports property for other namespaces. Imported namespaces are not transitive, and circular imports are possible.
  • Independence: Namespaces are not inherited or embedded; child namespaces are independent, requiring an import or fully qualified name for references.
  • Implicit root namespace: Contains any modules defined outside all explicit namespaces and is implicitly imported by all other namespaces.
  • Name Lookups:
    • Short module name: Used for modules within the same namespace or in an imported namespace.
    • Fully-qualified name: Used for modules outside the current namespace, in the form //path/to/module:module_name.
    • ModuleFromName lookups starting with // use the fully-qualified name map; otherwise, nameData is used to find current and imported namespaces, checking the short name map.
  • Core platform code: All core platform code will reside in a single root namespace and can continue to use short names.
  • Implicit module dependencies: Always need to be fully-qualified names to ensure accessibility from any namespace.
  • Export to Android.mk: Module names must be unique when exported to Android.mk. Products select which Soong namespaces they need, and only modules from selected namespaces will be exported. Modules in unselected namespaces are still built as part of make checkbuild.
  • Implementation Details:
    • Blueprint adds a NameInterface allowing Soong to map names to blueprint.Module instances.
    • Soong's NameInterface implementation stores maps for module lookup.
    • Each module gets a “fully-qualified name” (// + ctx.ModuleDir() + : + ModuleName()).
    • Each module also gets a “short name” stored as a (namespace, name) tuple. Both fully-qualified and short names are used as keys in a map to the blueprint.Module.
    • When a Namespace type module is encountered, Soong uses ctx.SetNameData() to store the new namespace, affecting subsequent modules.

4.4. Future Improvements and Caveats

  • Visibility rules: Could be implemented using a soong_visibility module, enforced at name lookup or a later check stage.
  • Name conflicts: Semantics for name conflicts in imported namespaces need to be defined, likely requiring uniqueness across imported namespaces for Android.mk export.