WARNING: SessionBundle has been deprecated and is no longer supported. Switch to SavedModel immediately.
This document describes the data formats and layouts for exporting TensorFlow models for inference.
These exports have the following properties:
The TensorFlow Saver
writes checkpoints (graph variables) while training so it can recover if it crashes. A TensorFlow Serving export contains a checkpoint with the current state of the graph variables along with a MetaGraph definition that's needed for serving.
# Directory overview 00000000/ assets/ export.meta export-?????-of-?????
00000000
-- Export version%08d
assets
-- Asset file directoryexport.meta
-- MetaGraph Definitiontensorflow::MetaGraphDef
export-?????-of-?????
Saver
with sharded=True
.The Exporter
class can be used to export a model in the above format from a TensorFlow Python binary.
TF.learn uses an Exporter wrapper that can be used for building signatures. Use the BaseEstimator.export
function to export your Estimator with a signature.
The LoadSessionBundleFromPath
function can be used to create a tensorflow::Session
and initialize it from an export. This function takes session options and the path to the export as arguments and returns a bundle of export data including a tensorflow::Session
which can be run.
Graphs used for inference tasks typically have set of inputs and outputs used at inference time. We call this a ‘Signature’.
Graphs used for standard inference tasks have standard sets of inputs and outputs. For example, a graph used for a regression task has an input tensor for the data and an output tensor for the regression values. The signature mechanism makes it easy to identify the relevant input and output tensors for common graph applications.
The Manifest
contains a Signature
message which contains the task specific inputs and outputs.
// A Signature specifies the inputs and outputs of commonly used graphs. message Signature { oneof type { RegressionSignature regression_signature = 1; ClassificationSignature classification_signature = 2; GenericSignature generic_signature = 3; } };
A Standard Signature can be set at export time using the Exporter
API.
# Run an export. signature = exporter.classification_signature(input_tensor=input, classes_tensor=output) export = exporter.Exporter(saver) export.init(sess.graph.as_graph_def(), default_graph_signature=signature) export.export(export_path, global_step_tensor, sess)
TF.learn models can use the BaseEstimator.export
function directly to export. To specify a Signature, use the Exporter wrapper helpers (e.g. classification_signature_fn
).
estimator = tf.contrib.learn.Estimator(...) ... # Other possible parameters omitted for the sake of brevity. estimator.export( export_path, signature_fn=tf.contrib.learn.utils.export.classification_signature_fn)
These can be recovered at serving time using utilities in signature.h
// Get the classification signature. ClassificationSignature signature; TF_CHECK_OK(GetClassificationSignature(bundle->meta_graph_def, &signature)); // Run the graph. Tensor input_tensor = GetInputTensor(); Tensor classes_tensor; Tensor scores_tensor; TF_CHECK_OK(RunClassification(signature, input_tensor, session, &classes_tensor, &scores_tensor));
Generic Signatures enable fully custom usage of the tensorflow::Session
API. They are recommended for when the Standard Signatures do not satisfy a particular use-case. A general example of when to use these is for a model taking a single input and generating multiple outputs performing different inferences.
// GenericSignature specifies a map from logical name to Tensor name. // Typical application of GenericSignature is to use a single GenericSignature // that includes all of the Tensor nodes and target names that may be useful at // serving, analysis or debugging time. The recommended name for this signature // is "generic_bindings". message GenericSignature { map<string, TensorBinding> map = 1; };
Generic Signatures can be used to compliment a Standard Signature, for example to support debugging. Here is an example that includes the Standard regression Signature and a Generic Signature.
named_tensor_bindings = {"logical_input_A": v0, "logical_input_B": v1} signatures = { "regression": exporter.regression_signature(input_tensor=v0, output_tensor=v1), "generic": exporter.generic_signature(named_tensor_bindings)} export = exporter.Exporter(saver) export.init(sess.graph.as_graph_def(), named_graph_signatures=signatures) export.export(export_path, global_step_tensor, sess)
Generic Signature does not differentiate between input and output tensors. It provides full flexibility to specify the input and output tensors you need. The benefit is preserving a mapping between names that you specify at export time (we call these the logical names), and the actual graph node names that may be less stable and/or auto-generated by TensorFlow.
In signature.h
note that the generic signature methods BindGenericInputs
and BindGenericNames
are doing simple string to string mapping as a convenience. These methods map from the names used at training time to actual names in the graph.
The bound results from those methods can be used as inputs to tensorflow::Session->Run()
. Specifically, the bound result vector<pair<string, Tensor>>
from BindGenericInputs
can be supplied as the first parameter inputs
to tensorflow::Session->Run()
. Similarly, the bound result vector<string>
from BindGenericNames
, can be mapped to output_tensor_names
in the tensorflow::Session->Run()
arguments. The next parameter, target_node_names
is typically null at inference time. The last parameter outputs
is for the results, which share the same order as the supplied output_tensor_names
.
Some graphs many require custom initialization after the variables have been restored. Such initialization, done through an arbitrary Op, can be added using the Exporter
API. If set, LoadSessionBundleFromPath
will automatically run the Op when restoring a Session
following the loading of variables.
In many cases we have Ops which depend on external files for initialization (such as vocabularies). These “assets” are not stored in the graph and are needed for both training and inference.
In order to create hermetic exports these asset files need to be:
Copying assets to the export directory is handled with a callback mechanism. The callback function receives two parameters:
gfile.Copy
to perform the copy.The tensor that contains the filepath to be copied is specified by passing the collection of asset filepath tensor, which is usually extracted from the graph by tf.get_collection(tf.GraphKeys.ASSET_FILEPATHS)
.
# Run an export. export = exporter.Exporter(save) export.init( sess.graph.as_graph_def(), asset_collection=tf.get_collection(tf.GraphKeys.ASSET_FILEPATHS)) export.export(export_path, global_step_tensor, sess)
Users can use their own callbacks as shown in the following example, with the requirement to keep the basename of the original files:
def my_custom_copy_callback(files_to_copy, export_dir_path): # Copy all source files (keys) in files_to_copy to export_dir_path using the # corresponding basename (value). ... # Run an export. export = exporter.Exporter(save) export.init( sess.graph.as_graph_def(), asset_collection=tf.get_collection(tf.GraphKeys.ASSET_FILEPATHS), asset_callback=my_custom_copy_callback) export.export(export_path, global_step_tensor, sess)
AssetFile
binds the name of a tensor in the graph to the name of a file within the assets directory. LoadSessionBundleFromPath
will handle the base path and asset directory swap/concatenation such that the tensor is set with the fully qualified filename upon return.
The typical workflow of model exporting is:
The Exporter should be used as follows: