tree ad27e92b88d824842bc9830e0ac98b2e6a9fa19e
parent 3125e6a821f00c402d6cd2b4beae9f9eb731ebee
author Aapo Kyrola <akyrola@fb.com> 1480633524 -0800
committer Bram Wasti <bwasti@dev11999.prn1.facebook.com> 1480967606 -0800

Fix consequtive checkpoint syncs

Summary: Switching to Pieter-MPI changed the way we setup network between operators. For syncronizing parameters after a checkpoint load, we run a checkpoint_net that contaiend operators for creating the common world and broadcast operators. Unfortunately this fails when the checkpoint sync is done a second time, because we would have created a duplicate common world. Solution is to separate common world op and broadcast op to init net and the actual broadcasting net, and we run the init net only once. This problem did not arise in the Flow version since I did only one checkpoint loading per operator (process).

Differential Revision: D4251754

fbshipit-source-id: ba030579e651e529e29bbf2d27920075078d8ff9