Minion upload after purge may leave segment in an inconsistent state

This is a race condition that we have observed during the following sequence of events.
- Minion purge task downloads a segment and purges some records
- The task tries to upload the segment back to the controller via controller API
- The controller happens to field a restart while moving the file to deepstore (we use LocalFS, but this may happen in other FS as well, not sure).

In this case, it is possible that the deep store does not have a copy of the segment, but the segment exists in idealstate, and the segment metadata exists as well.

We have not worked out the details of the exact sub-steps during the upload process that lets the segment reach this state. It could be because LocalFS first deletes the segment in deep store, and then copies the new segment from local file over to deep store, and the restart happens between these two operations.

Also, not sure if minion retries the push, but even if it does so, it probably fails because the segment is in inconsistent state (or because concurrent push is enabled).

We will be digging more into this, and proposing fixes.

Start by tracing the minion purge upload through the controller API and LocalFS deep-store move, then inspect how idealstate and segment metadata are updated around restart and retry. Done means reproducing or explaining the race and implementing a recovery path that prevents or repairs a segment present in idealstate but absent from deep store.

Written by the indexing model from the issue text.