tree f9416951c7e1f657cf5d93d8473924c1290f3d1c
parent 0330f67b222bb58dcc3b47941338086e7538e704
author Xiaodong Wang <xdwang@meta.com> 1687485473 +0000
committer PyTorch MergeBot <pytorchmergebot@users.noreply.github.com> 1687485473 +0000

[nccl] Do no skip send/recv 0 byte tensor (#103140)

Summary:
Since NCCL 2.12.10, NCCL supports send/recv 0 byte: https://github.com/NVIDIA/nccl/issues/696. Therefore we don't have to skip.

One issue is that if a rank has 0 bytes to send and 0 bytes to recv, it'll skip send/recv completely. And it'll proceed to the next collective which it can send/recv something, making it confusing to the other ranks. Another solution is to add a barrier but that's very expensive.

Test Plan: will add a unit test

Differential Revision: D46507785

Pull Request resolved: https://github.com/pytorch/pytorch/pull/103140
Approved by: https://github.com/malfet, https://github.com/kwen2501