This document explains how to use the TensorFlow Lite Hexagon Delegate in your application using the Java and/or C API. The delegate leverages the Qualcomm Hexagon library to execute quantized kernels on the DSP. Note that the delegate is intended to complement NNAPI functionality, particularly for devices where NNAPI DSP acceleration is unavailable (e.g., on older devices, or devices that don’t yet have a DSP NNAPI driver).
Note: This delegate is in experimental (beta) phase.
Supported devices:
Currently the following Hexagon architecture are supported, including but not limited to:
Supported models:
The Hexagon delegate supports all models that conform to our 8-bit symmetric quantization spec, including those generated using post-training integer quantization. UInt8 models trained with the legacy quantization-aware training path are also supported, for e.g., these quantized versions on our Hosted Models page.
public class HexagonDelegate implements Delegate, Closeable { /* * Creates a new HexagonDelegate object given the current 'context'. * Throws UnsupportedOperationException if Hexagon DSP delegation is not * available on this device. */ public HexagonDelegate(Context context) throws UnsupportedOperationException /** * Frees TFLite resources in C runtime. * * User is expected to call this method explicitly. */ @Override public void close(); }
dependencies { ... implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT' implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT' }
Note: You will need to accept the license agreement.
Note: As of 07/22/2020 you should use v1.20.
Note: You must use the hexagon_nn libraries with the compatible version of interface library. Interface library is part of the AAR and fetched by bazel through the config The version in the bazel config is the version you should use.
Note: If your app will be built for both 32 and 64-bit ARM devices, then you will need to add the Hexagon shared libs to both 32 and 64-bit lib folders.
import org.tensorflow.lite.HexagonDelegate; // Create the Delegate instance. try { hexagonDelegate = new HexagonDelegate(activity); tfliteOptions.addDelegate(hexagonDelegate); } catch (UnsupportedOperationException e) { // Hexagon delegate is not supported on this device. } tfliteInterpreter = new Interpreter(tfliteModel, tfliteOptions); // Dispose after finished with inference. tfliteInterpreter.close(); if (hexagonDelegate != null) { hexagonDelegate.close(); }
struct TfLiteHexagonDelegateOptions { // This corresponds to the debug level in the Hexagon SDK. 0 (default) // means no debug. int debug_level; // This corresponds to powersave_level in the Hexagon SDK. // where 0 (default) means high performance which means more power // consumption. int powersave_level; // If set to true, performance information about the graph will be dumped // to Standard output, this includes cpu cycles. // WARNING: Experimental and subject to change anytime. bool print_graph_profile; // If set to true, graph structure will be dumped to Standard output. // This is usually beneficial to see what actual nodes executed on // the DSP. Combining with 'debug_level' more information will be printed. // WARNING: Experimental and subject to change anytime. bool print_graph_debug; }; // Return a delegate that uses Hexagon SDK for ops execution. // Must outlive the interpreter. TfLiteDelegate* TfLiteHexagonDelegateCreate(const TfLiteHexagonDelegateOptions* options); // Do any needed cleanup and delete 'delegate'. void TfLiteHexagonDelegateDelete(TfLiteDelegate* delegate); // Initializes the DSP connection. // This should be called before doing any usage of the delegate. // "lib_directory_path": Path to the directory which holds the // shared libraries for the Hexagon NN libraries on the device. void TfLiteHexagonInitWithPath(const char* lib_directory_path); // Same as above method but doesn't accept the path params. // Assumes the environment setup is already done. Only initialize Hexagon. Void TfLiteHexagonInit(); // Clean up and switch off the DSP connection. // This should be called after all processing is done and delegate is deleted. Void TfLiteHexagonTearDown();
dependencies { ... implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT' implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT' }
Note: You will need to accept the license agreement.
Note: As of 07/22/2020 you should use v1.20.
Note: You must use the hexagon_nn libraries with the compatible version of interface library. Interface library is part of the AAR and fetched by bazel through the config. The version in the bazel config is the version you should use.
Note: If your app will be built for both 32 and 64-bit ARM devices, then you will need to add the Hexagon shared libs to both 32 and 64-bit lib folders.
In your code, ensure the native Hexagon library is loaded. This can be done by calling System.loadLibrary("tensorflowlite_hexagon_jni");
in your Activity or Java entry-point.
Create a delegate, example:
#include "tensorflow/lite/delegates/hexagon/hexagon_delegate.h" // Assuming shared libraries are under "/data/local/tmp/" // If files are packaged with native lib in android App then it // will typically be equivalent to the path provided by // "getContext().getApplicationInfo().nativeLibraryDir" const char[] library_directory_path = "/data/local/tmp/"; TfLiteHexagonInitWithPath(library_directory_path); // Needed once at startup. ::tflite::TfLiteHexagonDelegateOptions params = {0}; // 'delegate_ptr' Need to outlive the interpreter. For example, // If use case will need to resize input or anything that can trigger // re-applying delegates then 'delegate_ptr' need to outlive the interpreter. auto* delegate_ptr = ::tflite::TfLiteHexagonDelegateCreate(¶ms); Interpreter::TfLiteDelegatePtr delegate(delegate_ptr, [](TfLiteDelegate* delegate) { ::tflite::TfLiteHexagonDelegateDelete(delegate); }); interpreter->ModifyGraphWithDelegate(delegate.get()); // After usage of delegate. TfLiteHexagonTearDown(); // Needed once at end of app/DSP usage.
app/src/main/jniLibs/arm64-v8a
app/src/main/jniLibs/armeabi-v7a
Note: If you're using App Bundle for publishing your Application, you might want to set android.bundle.enableUncompressedNativeLibs=false in the gradle.properties file.
For issues, please create a GitHub issue with all the necessary repro details, including the phone model and board used (adb shell getprop ro.product.device
and adb shell getprop ro.board.platform
).
Created TensorFlow Lite delegate for Hexagon.
Hexagon delegate: X nodes delegated out of Y nodes.
bazel build -c opt --config=android_arm64 tensorflow/lite/delegates/hexagon/java:tensorflow-lite-hexagon
.adb shell cat /proc/cpuinfo | grep Hardware
and see if it returns something like “Hardware : Qualcomm Technologies, Inc MSMXXXX”.adb shell setenforce 0