API File Generation

There are certain pieces of NeuralNetworksTypes.h, Types.h, OperandTypes.h, OperationTypes.h, and of our various *.hal files that ought to be kept in sync -- most notably the operand type and operation type definitions and descriptions. To avoid having to do this manually, a tool generate_api.py is employed to combine a single specification file with one template file per API file (NeuralNetworksTypes.h, Types.h, OperandTypes.h, OperationTypes.h, or types.hal) to produce that API file. The script generate_api.sh invokes generate_api.py once per API file, passing appropriate arguments.

generate_api.sh

The environment variable ANDROID_BUILD_TOP must be set.

Invoked with no arguments or with the --mode=update argument, this script regenerates each API file in place, by invoking generate_api.py once per generated file.

Invoked with the --mode=hook argument, this script checks whether NeuralNetworksTypes.h, Types.h, OperandTypes.h, or OperationTypes.h needs to be regenerated.

When the --dryrun argument is present, this script shows how it would invoke generate_api.py but does not actually regenerate files or check whether they need to be regenerated.

generate_api.py

This tool generates a single output file from an input specification file and an input template file. It takes the following mandatory arguments:

• --output OUTPUT path to generated output file (such as Types.h)
• --specification SPECIFICATION path to input specification file
• --template TEMPLATE path to input template file
• --kind KIND token identifying kind of file to generate

The “kind” is an arbitrary token that the specification file can reference with the %kind directive to help generate different text in different situations. It has no meaning to the tool itself. Today, the following kinds are used: ndk (when generating NeuralNetworksTypes.h), canonical (when generating Types.h, OperandTypes.h, and OperationTypes.h), hal_1.0 (when generating 1.0/types.hal), hal_1.1, hal_1.2, and hal_1.3.

Template File Syntax

Every line of the template file is copied verbatim to the output file unless that line begins with %.

A line that begins with %% is a comment, and is ignored.

A line that begins with % and is not a comment is a directive.

Directives

%insert *name*

Copy the section with the specified name from the specification file to the output file. The section is defined by a %section directive in the specification file.

%insert-indented *count* *name*

Similar to %insert *name*, but each non-empty copied line is prefixed with count space characters. count must be a non-negative integer.

Specification File Syntax

The specification file consists of comments, directives, and other text.

A line that begins with %% is a comment, and is ignored.

A line that begins with % and is not a comment is a directive.

The meaning of a line that is neither a comment nor a directive depends on the context -- the region in which that line appears.

Regions

The specification file is divided into regions, which are sequences of lines delimited by certain directives.

Certain regions can enclose certain other regions, but this is very limited:

• A conditional region can enclose a section region.
• A section region can enclose a conditional region.

Equivalently:

• A conditional region can be enclosed by a section region.
• A section region can be enclosed by a conditional region.

null region

A null region is a sequence of lines that is not part of any other region. For example, a specification file that contains no directives other than %define and %define-kinds consists of a single null region.

Within a null region, all lines other than directives are treated as comments and are ignored.

conditional region

A conditional region is a sequence of lines immediately preceded by the %kind *list* directive and immediately followed by the %/kind directive. The %kind directive establishes a condition state on or off (see the description of the directive for details). When the condition is on, the lines in the region are processed normally (i.e., directives have their usual effect, and non-directive lines are added to the enclosing section region, if any). When the condition is off, lines in the region other than the %else directive are ignored except that even ignored directives undergo some level of syntactic and semantic checking.

section region

A section region is a sequence of lines immediately preceded by the %section *name* directive and immediately followed by the %/section directive. Every line in the sequence that doesn‘t begin with % undergoes macro substitution, and the resulting lines are associated with the section name. They can be inserted into the generated output file as directed by the template file’s %insert and %insert-indented directives. They can be added to another section region with the with the specification file's %insert and %insert-indented directives.

This is the mechanism by which a specification file contributes text to the generated output file.

Directives

%define *name* *body*

Defines a macro identified by the token name. The body is separated from the name by exactly one whitespace character, and extends to the end of the line -- it may contain whitespace itself. For example,

%define test this body begins and ends with a space character

Macro substitution occurs within a section region: a substring %{*name*} is replaced with the corresponding body. Macro substitution is not recursive: A substring %{*name2*} in body will not undergo macro substitution, except as discussed for macro arguments below.

Permitted in regions: null, conditional, section

macro arguments

The more general form of a macro invocation is %{*name* *arglist*}, where arglist is a list of whitespace-separated arguments. Within the body, a substring of the form %{argnum} will be replaced by the corresponding argument from arglist. For example, if the definition is

%define test second is %{2}, first is %{1}

then the macro invocation

%{test alpha beta}

is expanded to

second is beta, first is alpha

The only check on the number of arguments supplied at macro invocation time is that there must be at least as many arguments as the highest %{argnum} reference in the macro body. In the example above, %{test alpha} would be an error, but %{test alpha beta gamma} would not.

%insert *name*

Adds all lines from the named section region to the current section region.

Permitted in regions: section

%insert-indented *count* *name*

Similar to %insert *name*, but each non-empty added line is prefixed with count space characters. count must be a non-negative integer.

Permitted in regions: section

%kind *list*, %else, %/kind

%kind *list* creates a conditional region terminated by %/kind.

The list consists of a space-delimited list of tokens, any of which may end in * to indicate a wildcard pattern or + to indicate a lowest version pattern. Any other pattern is a simple pattern. The condition is on in three cases:

• One of the simple pattern tokens equals the “kind”
• One of the wildcard pattern tokens less the * is a prefix of the “kind”
• One of the lowest version pattern tokens less the + matches the “kind” or the “kind” matches any token to the right from the lowest version pattern in the list passed to %define-kinds

In all other cases, the condition is off.

Within the region, the condition is inverted every time the %else directive appears.

Permitted in regions: null, section

%define-kinds *list*

This directive has two purposes:

• Validity-checking. If the “kind” is not on the space-delimited list of tokens, generate_api.py terminates with an error.
• Ordering the possible kinds for the lowest version pattern (see the section above for the explanation of the pattern).

Only one such directive is allowed per specification file.

Permitted in regions: null, section

%section *name*, %/section

%section *name* creates a section region terminated by %/section.

Permitted in regions: null, conditional