You should create new test specs in runtime/test/specs/<version>/ and name it with .mod.py suffix, so that other tools can automatically update the unit tests.
OperandType(name, (type, shape, <optional scale, zero point>), <optional initializer>)
For example,
# p1 is a 2-by-2 fp matrix parameter, with value [1, 2; 3, 4] p1 = Parameter("param", ("TENSOR_FLOAT32", [2, 2]), [1, 2, 3, 4]) # i1 is a quantized input of shape (2, 256, 256, 3), with scale = 0.5, zero point = 128 i1 = Input("input", ("TENSOR_QUANT8_ASYMM", [2, 256, 256, 3], 0.5, 128)) # p2 is an Int32 scalar with value 1 p2 = Int32Scalar("act", 1)
There are currently 10 operand types supported by the test generator.
# Instantiate a model model = Model() # Instantiate a model with a name model2 = Model("model_name")
model.Operation(optype, i1, i2, ...).To(o1, o2, ...)
For example,
model.Operation("ADD", i1, i2, act).To(o1)
Simple scalar and 1-D vector parameters can now be directly passed to Operation constructor, and test generator will deduce the operand type from the value provided.
model.Operation("MEAN", i1, [1], 0) # axis = [1], keep_dims = 0
Note that, for fp values, the initializer should all be Python fp numbers, e.g. use 1.0 or 1. instead of 1 for implicit fp operands.
The combination of inputs and expected outputs is called an example for a given model. An example is defined like
# Example 1, separate dictionary for inputs and outputs input1 = { i1: [1, 2], i2: [3, 4] } output1 = {o1: [4, 6]} # Example 2, combined dictionary example2_values = { i1: [5, 6], i2: [7, 8], o1: [12, 14] } # Instantiate an example Example((input1, output1), example2_values)
By default, examples will be attached to the most recent instantiated model. You can explicitly specify the target model, and optionally, the example name by
Example((input1, output1), example2_values, model=model, name="example_name")
You can add variations to the example so that the test generator can automatically create multiple tests. The following variations are supported:
Convert input/parameter/output to the specified type, e.g. float32 -> quant8. The target data type for each operand to transform has to be explicitly specified. It is the spec writer's responsibility to ensure such conversion is valid.
converter = DataTypeConverter(name="variation_name").Identify({ op1: (target_type, target_scale, target_zero_point), op2: (target_type, target_scale, target_zero_point), ... })
Convert input/parameter/output between NHWC and NCHW. The caller need to provide a list of target operands to transform, and also the data layout parameter to set.
converter = DataLayoutConverter(target_data_layout, name="variation_name").Identify( [op1, op2, ..., layout_parameter] )
Transpose a certain axis in input/output to target position, and optionally remove some axis. The caller need to provide a list of target operands to transform, and also the axis parameter to set.
converter = AxisConverter(originalAxis, targetAxis, dimension, drop=[], name="variation_name").Identify( [op1, op2, ..., axis_parameter] )
This model variation is for ops that apply calculation along certain axis, such as L2_NORMALIZATION, SOFTMAX, and CHANNEL_SHUFFLE. For example, consider L2_NORMALIZATION with input of shape [2, 3, 4, 5] along the last axis, i.e. axis = -1. The output shape would be the same as input. We can create a new model which will do the calculation along axis 0 by transposing input and output shape to [5, 2, 3, 4] and modify the axis parameter to 0. Such converter can be defined as
toAxis0 = AxisConverter(-1, 0, 4).Identify([input, output, axis])
The target axis can also be negative to test the negative indexing
toAxis0 = AxisConverter(-1, -4, 4).Identify([input, output, axis])
Consider the same L2_NORMALIZATION example, we can also create a new model with input/output of 2D shape [4, 5] by removing the first two dimension. This is essentially doing new_input = input[0,0,:,:] in numpy. Such converter can be defined as
toDim2 = AxisConverter(-1, -1, 4, drop=[0, 1]).Identify([input, output, axis])
If transposition and removal are specified at the same time, the converter will do transposition first and then remove the axis. For example, the following converter will result in shape [5, 4] and axis 0.
toDim2Axis0 = AxisConverter(-1, 2, 4, drop=[0, 1]).Identify([input, output, axis])
Convert the model to enable/disable relaxed computation.
converter = RelaxedModeConverter(is_relaxed, name="variation_name")
Convert the output by certain activation, the original activation is assumed to be NONE. The caller need to provide a list of target operands to transform, and also the activation parameter to set.
converter = ActivationConverter(name="variation_name").Identify( [op1, op2, ..., act_parameter] )
Add a placeholder ADD operation after each model output to make it as an internal operand. Will skip if the model does not have any output tensor that is compatible with the ADD operation or if the model has more than one operation.
Each example can have multiple groups of variations, and if so, will take the cartesian product of the groups. For example, suppose we declare a model with two groups, and each group has two variations: [[default, nchw], [default, relaxed, quant8]]. This will result in 6 examples: [default, default], [default, relaxed], [default, quant8], [nchw, default], [nchw, relaxed], [nchw, quant8].
Use AddVariations to add a group of variations to the example
# Add two groups of variations [default, nchw] and [default, relaxed, quant8] example.AddVariations(nchw).AddVariations(relaxed, quant8)
By default, when you add a group of variation, a unnamed default variation will be automatically included in the list. You can name the default variation by
example.AddVariations(nchw, defaultName="nhwc").AddVariations(relaxed, quant8)
Also, you can choose not to include default by
# Add two groups of variations [nchw] and [default, relaxed, quant8] example.AddVariations(nchw, includeDefault=False).AddVariations(relaxed, quant8)
The example above will result in 3 examples: [nchw, default], [nchw, relaxed], [nchw, quant8].
By default, the test generator will apply the following variations automatically.
AllTensorsAsInputsConverter: Convert all constant tensors in the model to model inputs. Will skip if the model does not have any constant tensor, or if the model has more than one operations. If not explicitly disabled, this variation will be automatically applied to all tests.
AllInputsAsInternalCoverter: Add a placeholder ADD operation before each model input to make it as an internal operand. Will skip if the model does not have any input tensor that is compatible to the ADD operation, or if the model has more than one operations. If not explicitly disabled, this variation will be automatically applied to all tests.
You can opt-out by invoking the corresponding methods on examples.
# Disable AllTensorsAsInputsConverter and AllInputsAsInternalCoverter. example.DisableLifeTimeVariation()
You may also specify a certain operand to be input/const-only that AllInputsAsInternalCoverter will skip converting this operand.
# "hash" will be converted to a model input when applying AllTensorsAsInputsConverter, # but will be skipped when further applying AllInputsAsInternalCoverter. hash = Parameter("hash", "TENSOR_FLOAT32", "{1, 1}", [0.123]).ShouldNeverBeInternal()
The test generator provides several helper functions or shorthands to add commonly used group of variations.
# Each following group of statements are equivalent # DataTypeConverter example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_FLOAT16", ...})) example.AddVariations("float16") # will apply to every TENSOR_FLOAT32 operands example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_INT32", ...})) example.AddVariations("int32") # will apply to every TENSOR_FLOAT32 operands # DataLayoutConverter example.AddVariations(DataLayoutConverter("nchw").Identify(op_list)) example.AddVariations(("nchw", op_list)) example.AddNchw(*op_list) # AxisConverter # original axis and dim are deduced from the op_list example.AddVariations(*[AxisConverter(origin, t, dim).Identify(op_list) for t in targets]) example.AddAxis(targets, *op_list) example.AddVariations(*[ AxisConverter(origin, t, dim).Identify(op_list) for t in range(dim) ], includeDefault=False) example.AddAllPositiveAxis(*op_list) example.AddVariations(*[ AxisConverter(origin, t, dim).Identify(op_list) for t in range(-dim, dim) ], includeDefault=False) example.AddAllAxis(*op_list) drop = list(range(dim)) drop.pop(origin) example.AddVariations(*[ AxisConverter(origin, origin, dim, drop[0:(dim-i)]).Identify(op_list) for i in dims]) example.AddDims(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, origin, dim, drop[0:i]).Identify(op_list) for i in range(dim)]) example.AddAllDims(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, j, dim, range(i)).Identify(op_list) \ for i in range(dim) for j in range(i, dim) ], includeDefault=False) example.AddAllDimsAndPositiveAxis(dims, *op_list) example.AddVariations(*[ AxisConverter(origin, k, dim, range(i)).Identify(op_list) \ for i in range(dim) for j in range(i, dim) for k in [j, j - dim] ], includeDefault=False) example.AddAllDimsAndAxis(dims, *op_list) # RelaxedModeConverter example.Addvariations(RelaxedModeConverter(True)) example.AddVariations("relaxed") example.AddRelaxed() # ActivationConverter example.AddVariations(ActivationConverter("relu").Identify(op_list)) example.AddVariations(("relu", op_list)) example.AddRelu(*op_list) example.AddVariations( ActivationConverter("relu").Identify(op_list), ActivationConverter("relu1").Identify(op_list), ActivationConverter("relu6").Identify(op_list)) example.AddVariations( ("relu", op_list), ("relu1", op_list), ("relu6", op_list)) example.AddAllActivations(*op_list)
Converters that support nested control flow models accept the following syntax:
converter = DataTypeConverter().Identify({
...
subgraphOperand: DataTypeConverter().Identify({
...
}),
...
})
If not explicitly specified, the minimal required HAL version will be inferred from the path, e.g. the models defined in nn/runtime/test/specs/V1_0/add.mod.py will all have version V1_0. However there are several exceptions that a certain operation is under-tested in previous version and more tests are added in a later version. In this case, two methods are provided to set the version manually.
Use IntroducedIn to set the version of a model. All variations of the model will have the same version.
model_V1_0 = Model().IntroducedIn("V1_0") ... # All variations of model_V1_0 will have the same version V1_0. Example(example, model=model_V1_0).AddVariations(var0, var1, ...)
Use Example.SetVersion to override the model version for specific tests. The target tests are specified by names. This method can also override the version specified by IntroducedIn.
Example.SetVersion(<version>, testName0, testName1, ...)
This is useful when only a subset of variations has a different version.
Use Model.IdentifyInputs and Model.IdentifyOutputs to explicitly specify model inputs and outputs. This is particularly useful for models referenced by IF and WHILE operations.
DataType = ["TENSOR_INT32", [1]] BoolType = ["TENSOR_BOOL8", [1]] def MakeConditionModel(): a = Input("a", DataType) b = Input("b", DataType) out = Output("out", BoolType) model = Model() model.IdentifyInputs(a, b) # "a" is unused by the model. model.IdentifyOutputs(out) model.Operation("LESS", b, [10]).To(out) return model def MakeBodyModel(): a = Input("a", DataType) b = Input("b", DataType) a_out = Output("a_out", DataType) b_out = Output("b_out", DataType) model = Model() model.IdentifyInputs(a, b) # The order is the same as in the WHILE operation. model.IdentifyOutputs(a_out, b_out) model.Operation("SUB", b, a, 0).To(a_out) model.Operation("ADD", b, [1], 0).To(b_out) return model a = Input("a", DataType) a_out = Output("a_out", DataType) cond = MakeConditionModel() body = MakeBodyModel() b_init = [1] Model().Operation("WHILE", cond, body, a, b_init).To(a_out)
Negative test, also known as validation test, is a testing method that supplies invalid model or request, and expects the target framework or driver to fail gracefully. You can use ExpectFailure to tag a example as invalid.
Example.ExpectFailure()
# Declare input, output, and parameters i1 = Input("op1", ("TENSOR_FLOAT32", [1, 3, 4, 1])) f1 = Parameter("op2", ("TENSOR_FLOAT32", [1, 3, 3, 1]), [1, 4, 7, 2, 5, 8, 3, 6, 9]) b1 = Parameter("op3", ("TENSOR_FLOAT32", [1]), [-200]) act = Int32Scalar("act", 0) o1 = Output("op4", ("TENSOR_FLOAT32", [1, 3, 4, 1])) # Instantiate a model and add CONV_2D operation # Use implicit parameter for implicit padding and strides Model().Operation("CONV_2D", i1, f1, b1, 1, 1, 1, act, layout).To(o1) # Additional data type quant8 = DataTypeConverter().Identify({ i1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), f1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), b1: ("TENSOR_INT32", 0.25, 0), o1: ("TENSOR_QUANT8_ASYMM", 1.0, 50) }) # Instantiate an example example = Example({ i1: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], o1: [0, 0, 0, 0, 35, 112, 157, 0, 0, 34, 61, 0] }) # Only use NCHW data layout example.AddNchw(i1, f1, o1, layout, includeDefault=False) # Add two more groups of variations example.AddInput(f1, b1).AddVariations("relaxed", quant8).AddAllActivations(o1, act) # The following variations are added implicitly. # example.AddVariations(AllTensorsAsInputsConverter()) # example.AddVariations(AllInputsAsInternalCoverter()) # The following variation is added implicitly if this test is introduced in v1.2 or later. # example.AddVariations(DynamicOutputShapeConverter())
The spec above will result in 96 tests if introduced in v1.0 or v1.1, and 192 tests if introduced in v1.2 or later.
Once you have your model ready, run
$ANDROID_BUILD_TOP/packages/modules/NeuralNetworks/runtime/test/specs/generate_all_tests.sh
It will update all CTS and VTS tests based on spec files in nn/runtime/test/specs/V1_*/*.
Rebuild with mma afterwards.