CHRE API Parser + Code Generator for CHPP

Since one of the main use cases of CHPP is to support remoting a CHRE PAL implementation to a peripheral component, we leverage the CHRE PAL API definitions for the platform-specific implementation of those services, such as the WWAN service. However, the structures used in the CHRE PAL APIs are the same as those used in the CHRE API for nanoapps, which are not inherently serializable, as they include things like pointers to other structures. So we must create slightly updated versions of these to send over the wire with CHPP.

The chre_api_to_chpp.py script in this directory parses CHRE APIs according to instructions given in chre_api_annotations.json and generates structures and conversion code used for serialization and deserialization.

Serialization format

TL;DR: The CHPP format is just a packed and flattened version of the CHRE structures, replacing pointers with offsets.

The serialized output is meant to match the source CHRE structure as closely as possible, however some special handling is needed to be able to send them over the wire to a different processor in the system. The general process for generating a serializable CHPP version of a CHRE structure is as follows:

1. Rewrite the name to start with “Chpp” instead of “chre” and mark it packed, to ensure there's no padding between fields (which is compiler-dependent)
2. Flatten the structure by replacing pointers to nested structures/arrays with a ChppOffset which describes the location in the payload where the nested data resides.
3. A CHPP application layer header is allocated for convenience when used in CHPP. The header values need to be set by CHPP and they are not zeroed out. The header length is not included in the offset calculation.

ChppOffset is a collection of a 16-bit offset from the beginning of the payload, and a length. While we could implicitly derive the offset, the length cannot exceed 16 bits due to limitations in the encapsulating CHPP protocol (and we don't expect to bump up against this limit anyways), and including the offset field helps simplify decoding.

This approach allows for a highly optimized, in-place encoding and decoding that only requires converting between pointer and ChppOffset types, if the following conditions are met:

1. The size of the CHRE and CHPP structures are the same, and the offsets to all contained fields are the same, and the same property holds for all nested structures/unions
2. (Encoding only) All nested structures appear in contiguous memory - for example, if a structure has a pointer to an array, then the array appears immediately after the structure in memory, with no padding
3. (Decoding only) Processor alignment constraints are met for all fields, such that they can be safely accessed through normal means at the location in memory at which they reside, when accounting for any headers, etc.

If conditions 2 or 3 are not met, then optimized encode/decode is still possible after copying the memory to a separate allocation. So in-place conversion can't be done, but the only translation needed is between pointer and ChppOffset. If the first condition is not met, then conversion is done using field-wise assignment, which allows the compiler to generate the necessary instructions to handle differences in alignment and packing.

All generated code currently assumes that it's running on a little endian CPU, as the wire format requires little endian byte order.

TODO: The script currently only generates code for the pessimistic case, where none of the constraints are met. By generating code that checks sizeof/offsetof on all fields and placing the optimized path in an if/else condition, the compiler can evaluate condition 1 and prune away code for the less-optimal path.

Annotations

As the C API does not have enough information for the script to know how to handle things like variable-length arrays (VLAs), we use the chre_api_annotations.json file to provide this detail. The supported fields are explained in the illustrated example below:

[// A list of entries for each input file
{
  // Path starting at <android_root>/system/chre to the input file
  "filename": "chre_api/include/chre_api/chre/wwan.h",
  // List of #includes that also need to be parsed, e.g. because they provide
  // struct definitions that are used in the file
  "includes": [
    "chre_api/include/chre_api/chre/common.h"
  ],
  // Which files the output header should pull in to satisfy dependencies
  "output_includes": [
    "chpp/services/common_types.h",
    "chre_api/chre/wwan.h"
  ],
  // A list of entries providing additional information for structs/unions that
  // appear in the input
  "struct_info": [
    {
      "name": "chreWwanCellInfoResult",
      // List of annotations for fields within the struct
      "annotations": [
        // Instead of copying the input, always force setting a specific value
        // for a field. The value is whatever should appear in the code for the
        // assignment statement, or the value to memset it to for an array type.
        {
          // Which field within the struct this annotation applies to
          "field": "version",
          // The type of annotation we're supplying
          "annotation": "fixed_value",
          // Additional information is dependent upon the annotation type
          "value": "CHRE_WWAN_CELL_INFO_RESULT_VERSION"
        },
        // Since the 'cookie' field here is a void*, we're rewriting to a
        // uint32_t to keep the same structure size + field offsets (on 32-bit
        // architectures) - in practice we'll also force the value to 0
        {
          "field": "cookie",
          "annotation": "rewrite_type",
          "type_override": "uint32_t"
        },
        // Indicates a variable length array field, with the number of elements
        // given by another field in the same structure called cellInfoCount
        {
          "field": "cells",
          "annotation": "var_len_array",
          "length_field": "cellInfoCount"
        }
      ]
    },
    {
      "name": "chreWwanCellInfo",
      "annotations": [
        // In this case, we have a union field, where only one of the members is
        // used, according to the provided mapping on the adjacent discriminator
        // field
        {
          "field": "CellInfo",
          "annotation": "union_variant",
          "discriminator": "cellInfoType",
          "mapping": [
            ["CHRE_WWAN_CELL_INFO_TYPE_GSM", "gsm"],
            ["CHRE_WWAN_CELL_INFO_TYPE_CDMA", "cdma"],
            ["CHRE_WWAN_CELL_INFO_TYPE_LTE", "lte"],
            ["CHRE_WWAN_CELL_INFO_TYPE_WCDMA", "wcdma"],
            ["CHRE_WWAN_CELL_INFO_TYPE_TD_SCDMA", "tdscdma"],
            ["CHRE_WWAN_CELL_INFO_TYPE_NR", "nr"]
          ]
        }
      ]
    }
  ],
  // The list of top-level structures appearing in the input that we should
  // create conversion routines for
  "root_structs": [
    "chreWwanCellInfoResult"
  ]
}]

Requirements

Tested with Python 3.7, but most versions of Python 3 should work.

Requires pyclibrary - see requirements.txt.