tree cf04631df0aae4f8d92906a90c5aac075d304345
parent 26f7a8d6a9cb992f6498b4cf26188514e2f52a38
author Haoyu Zhang <haoyuzhang@google.com> 1575501520 -0800
committer TensorFlower Gardener <gardener@tensorflow.org> 1575501787 -0800

Protect EagerContext on worker side when updating cluster.

When handling worker failures, the failure handling thread sends update context request to all workers. In the meanwhile, other eager executors might be sending op/function execution requests. This change avoids the necessity of grabbing a global lock on the client side to prevent race conditions of concurrent updating and execution.

* Ref count the eager client to avoid deallocating them before pending requests finish.
* Hold context lock on worker side to avoid concurrently executing enqueue ops while handling context update.
* Adjust local device initialization to avoid clearing the _context_devices list since this can be called multiple times by update_server_def.

PiperOrigin-RevId: 283847202
Change-Id: I3f84d56c44cd2adce5136f7fd4f67313a1da3610