You should create new test specs in nn/runtime/test/specs/<version>/ and name it with .mod.py suffix, so that other tools can automatically update the unit tests.
OperandType(name, (type, shape, <optional scale, zero point>), <optional initializer>)
For example,
# p1 is a 2-by-2 fp matrix parameter, with value [1, 2; 3, 4] p1 = Parameter("param", ("TENSOR_FLOAT32", [2, 2]), [1, 2, 3, 4]) # i1 is a quantized input of shape (2, 256, 256, 3), with scale = 0.5, zero point = 128 i1 = Input("input", ("TENSOR_QUANT8_ASYMM", [2, 256, 256, 3], 0.5, 128)) # p2 is an Int32 scalar with value 1 p2 = Int32Scalar("act", 1)
There are currently 10 operand types supported by the test generator.
# Instantiate a model model = Model() # Instantiate a model with a name model2 = Model("model_name")
model.Operation(optype, i1, i2, ...).To(o1, o2, ...)
For example,
model.Operation("ADD", i1, i2, act).To(o1)
Simple scalar and 1-D vector parameters can now be directly passed to Operation constructor, and test generator will deduce the operand type from the value provided.
model.Operation("MEAN", i1, [1], 0) # axis = [1], keep_dims = 0
Note that, for fp values, the initializer should all be Python fp numbers, e.g. use 1.0 or 1. instead of 1 for implicit fp operands.
The combination of inputs and expected outputs is called an example for a given model. An example is defined like
# Example 1, separate dictionary for inputs and outputs input1 = { i1: [1, 2], i2: [3, 4] } output1 = {o1: [4, 6]} # Example 2, combined dictionary example2_values = { i1: [5, 6], i2: [7, 8], o1: [12, 14] } # Instantiate an example Example((input1, output1), example2_values)
By default, examples will be attached to the most recent instantiated model. You can explicitly specify the target model, and optionally, the example name by
Example((input1, output1), example2_values, model=model, name="example_name")
You can add variations to the example so that the test generator can automatically create multiple tests. Currently, 6 types of variation are supported:
Convert input/parameter/output to the specified type, e.g. float32 -> quant8. The target data type for each operand to transform has to be explicitly specified. It is the spec writer's responsibility to ensure such conversion is valid.
converter = DataTypeConverter(name="variation_name").Identify({ op1: (target_type, target_scale, target_zero_point), op2: (target_type, target_scale, target_zero_point), ... })
Convert input/parameter/output between NHWC and NCHW. The caller need to provide a list of target operands to transform, and also the data layout parameter to set.
converter = DataLayoutConverter(target_data_layout, name="variation_name").Identify( [op1, op2, ..., layout_parameter] )
Transpose a certain axis in input/output to target position, and optionally remove some axis. The caller need to provide a list of target operands to transform, and also the axis parameter to set.
converter = AxisConverter(originalAxis, targetAxis, dimension, drop=[], name="variation_name").Identify( [op1, op2, ..., axis_parameter] )
This model variation is for ops that apply calculation along certain axis, such as L2_NORMALIZATION, SOFTMAX, and CHANNEL_SHUFFLE. For example, consider L2_NORMALIZATION with input of shape [2, 3, 4, 5] along the last axis, i.e. axis = -1. The output shape would be the same as input. We can create a new model which will do the calculation along axis 0 by transposing input and output shape to [5, 2, 3, 4] and modify the axis parameter to 0. Such converter can be defined as
toAxis0 = AxisConverter(-1, 0, 4).Identify([input, output, axis])
The target axis can also be negative to test the negative indexing
toAxis0 = AxisConverter(-1, -4, 4).Identify([input, output, axis])
Consider the same L2_NORMALIZATION example, we can also create a new model with input/output of 2D shape [4, 5] by removing the first two dimension. This is essentially doing new_input = input[0,0,:,:] in numpy. Such converter can be defined as
toDim2 = AxisConverter(-1, -1, 4, drop=[0, 1]).Identify([input, output, axis])
If transposition and removal are specified at the same time, the converter will do transposition first and then remove the axis. For example, the following converter will result in shape [5, 4] and axis 0.
toDim2Axis0 = AxisConverter(-1, 2, 4, drop=[0, 1]).Identify([input, output, axis])
Convert the model to enable/disable relaxed computation.
converter = RelaxedModeConverter(is_relaxed, name="variation_name")
Convert a certain parameter to model input, e.g. weight in CONV_2D. The caller need to provide a list of target operands to convert.
converter = ParameterAsInputConverter(name="variation_name").Identify( [op1, op2, ...] )
Convert the output by certain activation, the original activation is assumed to be NONE. The caller need to provide a list of target operands to transform, and also the activation parameter to set.
converter = ActivationConverter(name="variation_name").Identify( [op1, op2, ..., act_parameter] )
Each example can have multiple groups of variations, and if so, will take the cartesian product of the groups. For example, suppose we declare a model with two groups, and each group has two variations: [[default, nchw], [default, relaxed, quant8]]. This will result in 6 examples: [default, default], [default, relaxed], [default, quant8], [nchw, default], [nchw, relaxed], [nchw, quant8].
Use AddVariations to add a group of variations to the example
# Add two groups of variations [default, nchw] and [default, relaxed, quant8] example.AddVariations(nchw).AddVariations(relaxed, quant8)
By default, when you add a group of variation, a unnamed default variation will be automatically included in the list. You can name the default variation by
example.AddVariations(nchw, defaultName="nhwc").AddVariations(relaxed, quant8)
Also, you can choose not to include default by
# Add two groups of variations [nchw] and [default, relaxed, quant8] example.AddVariations(nchw, includeDefault=False).AddVariations(relaxed, quant8)
The example above will result in 3 examples: [nchw, default], [nchw, relaxed], [nchw, quant8].
The test generator provides several helper functions or shorthands to add commonly used group of variations.
# Each following group of statements are equivalent # DataTypeConverter example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_FLOAT16", ...})) example.AddVariations("float16") # will apply to every TENSOR_FLOAT32 operands example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_INT32", ...})) example.AddVariations("int32") # will apply to every TENSOR_FLOAT32 operands # DataLayoutConverter example.AddVariations(DataLayoutConverter("nchw").Identify(op_list)) example.AddVariations(("nchw", op_list)) example.AddNchw(*op_list) # AxisConverter # original axis and dim are deduced from the op_list example.AddVariations(*[AxisConverter(origin, t, dim).Identify(op_list) for t in targets]) example.AddAxis(targets, *op_list) example.AddVariations(*[ AxisConverter(origin, t, dim).Identify(op_list) for t in range(dim) ], includeDefault=False) example.AddAllPositiveAxis(*op_list) example.AddVariations(*[ AxisConverter(origin, t, dim).Identify(op_list) for t in range(-dim, dim) ], includeDefault=False) example.AddAllAxis(*op_list) drop = list(range(dim)) drop.pop(origin) example.AddVariations(*[ AxisConverter(origin, origin, dim, drop[0:(dim-i)]).Identify(op_list) for i in dims]) example.AddDims(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, origin, dim, drop[0:i]).Identify(op_list) for i in range(dim)]) example.AddAllDims(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, j, dim, range(i)).Identify(op_list) \ for i in range(dim) for j in range(i, dim) ], includeDefault=False) example.AddAllDimsAndPositiveAxis(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, k, dim, range(i)).Identify(op_list) \ for i in range(dim) for j in range(i, dim) for k in [j, j - dim] ], includeDefault=False) example.AddAllDimsAndAxis(dims, *op_list) # ParameterAsInputConverter example.AddVariations(ParameterAsInputConverter().Identify(op_list)) example.AddVariations(("as_input", op_list)) example.AddInput(*op_list) # RelaxedModeConverter example.Addvariations(RelaxedModeConverter(True)) example.AddVariations("relaxed") example.AddRelaxed() # ActivationConverter example.AddVariations(ActivationConverter("relu").Identify(op_list)) example.AddVariations(("relu", op_list)) example.AddRelu(*op_list) example.AddVariations( ActivationConverter("relu").Identify(op_list), ActivationConverter("relu1").Identify(op_list), ActivationConverter("relu6").Identify(op_list)) example.AddVariations( ("relu", op_list), ("relu1", op_list), ("relu6", op_list)) example.AddAllActivations(*op_list)
If not explicitly specified, the minimal required HAL version will be inferred from the path, e.g. the models defined in nn/runtime/test/specs/V1_0/add.mod.py will all have version V1_0. However there are several exceptions that a certain operation is under-tested in previous version and more tests are added in a later version. In this case, two methods are provided to set the version manually.
Use IntroducedIn to set the version of a model. All variations of the model will have the same version.
model_V1_0 = Model().IntroducedIn("V1_0") ... # All variations of model_V1_0 will have the same version V1_0. Example(example, model=model_V1_0).AddVariations(var0, var1, ...)
Use Example.SetVersion to override the model version for specific tests. The target tests are specified by names. This method can also override the version specified by IntroducedIn.
Example.SetVersion(<version>, testName0, testName1, ...)
This is useful when only a subset of variations has a different version.
Negative test, also known as validation test, is a testing method that supplies invalid model or request, and expects the target framework or driver to fail gracefully. You can use ExpectFailure to tag a example as invalid.
Example.ExpectFailure()
# Declare input, output, and parameters i1 = Input("op1", "TENSOR_FLOAT32", "{1, 3, 4, 1}") f1 = Parameter("op2", "TENSOR_FLOAT32", "{1, 3, 3, 1}", [1, 4, 7, 2, 5, 8, 3, 6, 9]) b1 = Parameter("op3", "TENSOR_FLOAT32", "{1}", [-200]) act = Int32Scalar("act", 0) o1 = Output("op4", "TENSOR_FLOAT32", "{1, 3, 4, 1}") # Instantiate a model and add CONV_2D operation # Use implicit parameter for implicit padding and strides Model().Operation("CONV_2D", i1, f1, b1, 1, 1, 1, act, layout).To(o1) # Additional data type quant8 = DataTypeConverter().Identify({ i1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), f1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), b1: ("TENSOR_INT32", 0.25, 0), o1: ("TENSOR_QUANT8_ASYMM", 1.0, 50) }) # Instantiate an example example = Example({ i1: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], o1: [0, 0, 0, 0, 35, 112, 157, 0, 0, 34, 61, 0] }) # Only use NCHW data layout example.AddNchw(i1, f1, o1, layout, includeDefault=False) # Add two more groups of variations example.AddInput(f1, b1).AddVariations("relaxed", quant8).AddAllActivations(o1, act)
The spec above will result in 24 tests.
Once you have your model ready, run
$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh $ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_vts_test.sh
It will read and generate all CTS/VTS unit tests based on spec files in nn/runtime/test/specs/V1_*/* if needed. CTS test generator is able to identify which spec files are modified since last generation and only regenerate those files to reduce compilation time. To force a regeneration, use -f flag. The VTS test generator will regenerate tests targeting the latest HAL version by default. Pass the all positional argument to override.
Rebuild with mm afterwards.