tree bd5d5f3fcaaf4e599663b22d953f7a5992000858
parent 1f19aea3230acc158538951f4c510d98df3a99ee
author Chris Jones <cjfj@google.com> 1638376955 -0800
committer TensorFlower Gardener <gardener@tensorflow.org> 1638377266 -0800

[XLA:GPU] Add background thread that periodically checks for asynchronous NCCL errors.

NCCL can have asynchronous errors from, e.g., sockets. If an async error occurs, the GPU may deadlock. If the thread detects an asynchronous error, the communicator is aborted and an error logged.

PiperOrigin-RevId: 413425827
Change-Id: I7565c660895c7348792033d842c3efe7869e45af