tree 83116d324829176565095601900a59f4229a6cef
parent 7921c4d517dab0a17a6c24b309a04a12d4f71574
author Andrew Audibert <aaudibert@google.com> 1599066725 -0700
committer TensorFlower Gardener <gardener@tensorflow.org> 1599067008 -0700

Exchange device attributes at group resolution

Previously CollectiveParamResolver queries device attributes when initializing instance params. That has issues when the collective leader fails and restarts quickly between group resolution and instance resolution. In such case, all other workers get the incarnation of the restarted leader, thus they're unable to detect that the leader has failed; the leader will deadlock on the group resolution.

This change doesn't fully fixed the issue because it only exchanges device attributes at group resolution, but doesn't populate the device attributes to DeviceResolver. That will be done in a following change.

This change also changes the behavior when a non-leader fails and restarts. Previously it gets the cached group resolution from the leader, now it will get an error because its incarnation doesn't match with the one in the cached group parameters. This should have no actual effect since that worker will always restart again after the leader has restarted.

This change changes both the client and server without being backward compatible. It assumes that client and server are running the same version of Tensorflow. This should be true since the only way to use CollectiveParamResolverDistributed is through MultiWorkerMirroredStrategy (MWMS). For MWMS, all workers should run the same version of the program.

PiperOrigin-RevId: 329735919
Change-Id: I5c29a3ec8462c7737bcbbbf823a95693b0d27dc3