docs/source/quantization-accuracy-debugging.rst

Quantization Accuracy Debugging
-------------------------------

This document provides high level strategies for improving quantization
accuracy. If a quantized model has error compared to the original model,
we can categorize the error into:

1. **data insensitive error** - caused by intrinsic model quantization error,
   large portion of input data has large error
2. **data sensitive error** - caused by outlier input data, small
   portion of input data has large error
3. **implementation error** - quantized kernel is not matching reference implementation

Data insensitive error
~~~~~~~~~~~~~~~~~~~~~~

General tips
^^^^^^^^^^^^

1. For PTQ, ensure that the data you are calibrating with is representative
   of your dataset. For example, for a classification problem a general
   guideline is to have multiple samples in every category, and the overall
   number of samples should be at least 100. There is no penalty for
   calibrating with more data other than calibration time.
2. If your model has Conv-BN or Linear-BN patterns, consider fusing them.
   If you are using FX graph mode quantization, this is done automatically
   by the workflow. If you are using Eager mode quantization, you can do
   this manually with the ``torch.ao.quantization.fuse_modules`` API.
3. Increase the precision of dtype of the problematic ops. Usually, fp32
   will have the highest accuracy, followed by fp16, followed by dynamically
   quantized int8, followed by statically quantized int8.

   1. Note: this is trading off performance for accuracy.
   2. Note: availability of kernels per dtype per op can vary by backend.
   3. Note: dtype conversions add an additional performance cost. For example,
      ``fp32_op -> quant -> int8_op -> dequant -> fp32_op -> quant -> int8_op -> dequant``
      will have a performance penalty compared to
      ``fp32_op -> fp32_op -> quant -> int8_op -> int8_op -> dequant``
      because of a higher number of required dtype conversions.

4. If you are using PTQ, consider using QAT to recover some of the accuracy loss
   from quantization.

Int8 quantization tips
^^^^^^^^^^^^^^^^^^^^^^

1. If you are using per-tensor weight quantization, consider using per-channel
   weight quantization.
2. If you are doing inference on `fbgemm`, ensure that you set the `reduce_range`
   argument to `False` if your CPU is Cooperlake or newer, and to `True` otherwise.
3. Audit the input activation distribution variation across different samples.
   If this variation is high, the layer may be suitable for dynamic quantization
   but not static quantization.

Data sensitive error
~~~~~~~~~~~~~~~~~~~~

If you are using static quantization and a small portion of your input data is
resulting in high quantization error, you can try:

1. Adjust your calibration dataset to make it more representative of your
   inference dataset.
2. Manually inspect (using Numeric Suite) which layers have high quantization
   error. For these layers, consider leaving them in floating point or adjusting
   the observer settings to choose a better scale and zero_point.


Implementation error
~~~~~~~~~~~~~~~~~~~~

If you are using PyTorch quantization with your own backend
you may see differences between the reference implementation of an
operation (such as ``dequant -> op_fp32 -> quant``) and the quantized implementation
(such as `op_int8`) of the op on the target hardware. This could mean one of two things:

1. the differences (usually small) are expected due to specific behavior of
   the target kernel on the target hardware compared to fp32/cpu. An example of this
   is accumulating in an integer dtype. Unless the kernel guarantees bitwise
   equivalency with the reference implementation, this is expected.
2. the kernel on the target hardware has an accuracy issue. In this case, reach
   out to the kernel developer.

Numerical Debugging Tooling (prototype)
---------------------------------------

.. toctree::
    :hidden:

    torch.ao.ns._numeric_suite
    torch.ao.ns._numeric_suite_fx

.. warning ::
     Numerical debugging tooling is early prototype and subject to change.

* :ref:`torch_ao_ns_numeric_suite`
  Eager mode numeric suite
* :ref:`torch_ao_ns_numeric_suite_fx`
  FX numeric suite