Incomplete tasks due service crash

Often might happen that service crash due various reasons, for example pipeline-tarball might try to use more memory than available at node, then it will restart and we might have checkout node finished and stay in incomplete state.
We need a way to retry last job that was being executed, if it is possible.

Proposal: to do this we need mechanism that will take event, set this event as "current" of service, and on event completion - delete this state. In case service restarted, it will check if there was event that was being processed and if yes, it will retry to process it again.

Question is:
Where to store event? Persistent storage on kubernetes? some additional API endpoint?

Any other suggestions?

No files or tests are named. Start by tracing how the service processes events, including pipeline-tarball and checkout-node work, then compare persistent storage options available in Kubernetes. Done would mean an in-progress event survives a service restart, is retried, and is cleared after completion without duplicate recovery.

Written by the indexing model from the issue text.