commit 2c366a1fd06a42f0add9e94a15e3d65a370e25ee
author Lalit Maganti <lalitm@google.com> Sat Sep 02 23:35:55 2023 +0100
committer Lalit Maganti <lalitm@google.com> Sat Sep 02 23:35:55 2023 +0100
tree f8f3af07db8a013ce13c8c86e73dd34312ae8411
parent ab09049c909375a7998225d74adb2cae0e140282

ctp: overhaul worker API to make everything more resilient

This CL attempts to make CTP more resilient to workers restarting.
Specifically on Google infra, restarts of workers is fully expected
every few days and we need to be able to survive these.

The only real way to do this is to make all the worker state
ephemeral and make the orchestrator the source of truth. We do this
by moving to a "reconciliation" (i.e. Kubernetes) syle model. Concretely, the orchestrator holds onto all the state and pushes
this out to the workers every 15s. That way, even if a worker
restarts, it'll be in sync within some time.

This allows significant simplifications of the worker APIs as we no
longer need to track create/destroy etc only the presence/absence of
stateless traces. While doing this, also strip out all notion of
stateful trace processing which will need some serious thoughts on
designing this without losing state.

In the future, we might even make the manager stateless by moving that
state to a database system which can be accessed by all the components.
That way, the worker can pull the state rather than the manager pushing
it out. That again requires some significant thought and introduces yet
another component so we don't do that in this CL.

Change-Id: I8aadce37f86fbaddab59a512ceae61b9f7de755f
Bug: 278208757