ArmNN allows adding new backends through the ‘Pluggable Backend’ mechanism.
Backends reside under src/backends, in separate subfolders. For Linux builds they must have a backend.cmake
file which is read automatically by src/backends/backends.cmake. The backend.cmake
file under the backend-specific folder is then included by the main CMakeLists.txt file at the root of the ArmNN source tree.
The backend.cmake
has three main purposes:
armnnLibraries
list.armnnUnitTestLibraries
list.Example backend.cmake
file taken from reference/backend.cmake:
# # Make sure the reference backend is included in the build. # By adding the subdirectory, cmake requires the presence of CMakeLists.txt # in the reference (backend) folder. # add_subdirectory(${PROJECT_SOURCE_DIR}/src/backends/reference) # # Add the cmake OBJECT libraries built by the reference backend to the # list of libraries linked against the ArmNN shared library. # list(APPEND armnnLibraries armnnRefBackend armnnRefBackendWorkloads) # # Backend specific unit tests can be integrated through the # armnnUnitTestLibraries variable. This makes sure that the # UnitTests executable can run the backend-specific unit # tests. # list(APPEND armnnUnitTestLibraries armnnRefBackendUnitTests)
As described in the previous section, adding a new backend will require creating a CMakeLists.txt
in the backend folder. This follows the standard cmake conventions, and is required to build a static cmake OBJECT library to be linked into the ArmNN shared library. As with any cmake build, the code can be structured into subfolders and modules as the developer sees fit.
Example can be found under reference/CMakeLists.txt.
ArmNN on Android uses the native Android build system. New backends are integrated by creating a backend.mk
file which has a single variable called BACKEND_SOURCES
listing all cpp files to be built by the Android build system for the ArmNN shared library.
Optionally, backend-specific unit tests can be added similarly, by appending the list of cpp files to the BACKEND_TEST_SOURCES
variable.
Example taken from reference/backend.mk:
BACKEND_SOURCES := \ RefLayerSupport.cpp \ RefWorkloadFactory.cpp \ workloads/Activation.cpp \ workloads/ElementwiseFunction.cpp \ workloads/Broadcast.cpp \ ... BACKEND_TEST_SOURCES := \ test/RefCreateWorkloadTests.cpp \ test/RefEndToEndTests.cpp \ test/RefJsonPrinterTests.cpp \ ...
For multiple backends that need common code, there is support for including them in the build similarly to the backend code. This requires adding three files under a subfolder at the same level as the backends folders. These are:
They work the same way as the backend files. The only difference between them is that common code is built first, so the backend code can depend on them.
aclCommon is an example for this concept and you can find the corresponding files:
Backends are identified by a string that must be unique across backends. This string is wrapped in the BackendId object for backward compatibility with previous ArmNN versions.
All backends need to implement the IBackendInternal interface. The interface functions to be implemented are:
virtual IMemoryManagerUniquePtr CreateMemoryManager() const = 0; virtual IWorkloadFactoryPtr CreateWorkloadFactory( const IMemoryManagerSharedPtr& memoryManager = nullptr) const = 0; virtual IBackendContextPtr CreateBackendContext(const IRuntime::CreationOptions&) const = 0; virtual Optimizations GetOptimizations() const = 0; virtual ILayerSupportSharedPtr GetLayerSupport() const = 0; virtual SubGraphUniquePtr OptimizeSubGraph(const SubGraph& subGraph, bool& optimizationAttempted) const = 0;
Note that GetOptimizations()
has been deprecated. The method OptimizeSubGraph(...)
should be used instead to specific optimizations to a given sub-graph.
The ArmNN framework then creates instances of the IBackendInternal interface with the help of the BackendRegistry singleton.
Important: the IBackendInternal
object is not guaranteed to have a longer lifetime than the objects it creates. It is only intended to be a single entry point for the factory functions it has. The best use of this is to be a lightweight, stateless object and make no assumptions between its lifetime and the lifetime of the objects it creates.
For each backend one needs to register a factory function that can be retrieved using a BackendId. The ArmNN framework creates the backend interfaces dynamically when it sees fit and it keeps these objects for a short period of time. Examples:
GetLayerSupport()
function and creates an ILayerSupport
object to help deciding this.GetOptimizations()
function and it runs them on the network.IBackendContext
object and keeps this context alive for the Runtime's lifetime. It notifies this context object before and after a network is loaded or unloaded.As mentioned above, all backends need to be registered through the BackendRegistry so ArmNN knows about them. Registration requires a unique backend ID string and a lambda function that returns a unique pointer to the IBackendInternal interface.
For registering a backend only this lambda function needs to exist, not the actual backend. This allows dynamically creating the backend objects when they are needed.
The BackendRegistry has a few convenience functions, like we can query the registered backends and are able to tell if a given backend is registered or not.
ArmNN uses the ILayerSupport interface to decide if a layer with a set of parameters (i.e. input and output tensors, descriptor, weights, filter, kernel if any) are supported on a given backend. The backends need a way to communicate this information by implementing the GetLayerSupport()
function on the IBackendInternal
interface.
Examples of this can be found in the RefLayerSupport header and the RefLayerSupport implementation.
The IWorkloadFactory interface is used for creating the backend specific workloads. The factory function that creates the IWorkloadFactory object in the IBackendInterface takes an IMemoryManager object.
To create a workload object the IWorkloadFactory
takes a WorkloadInfo
object that holds the input and output tensor information and a workload specific queue descriptor.
Backends may choose to implement custom memory management. ArmNN supports this concept through the following mechanism:
IBackendInternal
interface has a CreateMemoryManager()
function which is called before creating the workload factoryCreateWorkloadFactory(...)
function so the workload factory can use it for creating the backend-specific workloadsAcquire()
on the memory manager before it starts executing the network and it calls Release()
in its destructorThe backends may choose to implement backend-specific optimizations. This is supported through the method OptimizeSubGraph(...)
to the backend interface that allows the backends to apply their specific optimizations to a given sub-grah.
The way backends had to provide a list optimizations to the Optimizer (through the GetOptimizations()
method) is still in place for backward compatibility, but it's now considered deprecated and will be remove in a future release.
Backends may need to be notified whenever a network is loaded or unloaded. To support that, one can implement the optional IBackendContext interface. The framework calls the CreateBackendContext(...)
method for each backend in the Runtime. If the backend returns a valid unique pointer to a backend context, then the runtime will hold this for its entire lifetime. It then calls the following interface functions for each stored context:
BeforeLoadNetwork(NetworkId networkId)
AfterLoadNetwork(NetworkId networkId)
BeforeUnloadNetwork(NetworkId networkId)
AfterUnloadNetwork(NetworkId networkId)