tree bbff092179928c6713e7a89cb54d815de52aaea1
parent f54290fd72e068472f92c9f37b23d095f9052c84
author Peter Bell <peterbell10@live.co.uk> 1625014270 -0700
committer Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> 1625014350 -0700

Accumulate 16-bit float sums in 32-bit accumulators (#60387)

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60387

Fixes gh-59489

Using 32-bit accumulators is a win-win: improved precision and improved
performance since the half precision types needed to be converted back and forth
to 32-bit float to do the arithmetic anyway.

Note that on multi-threaded or dis-contiguous sums, there can be partial sums
stored in the output so they are necessarily trucated to 16-bit. Fixing this
would require a rework of TensorIterator reductions.

Test Plan: Imported from OSS

Reviewed By: jbschlosser

Differential Revision: D29447187

Pulled By: ngimel

fbshipit-source-id: d0619e0ca2fe116d101460142b79ca56fd6d0840