FSDS compaction command can leave data in an intermediate state

FSDS compaction command can leave data in an intermediate state

The list of files that were compacted and need to be deleted is kept in memory, meaning if the driver program is killed then it never gets removed and the metadata never gets updated.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2162

Key: GEOMESA-2162
Type: Bug
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Mon, 29 Jan 2018 12:42:40 -0500
Updated: Thu, 13 Dec 2018 11:38:06 -0500
Affects Version(s): 2.0.0

Comments:

557058:48912df4-0004-4f90-9915-db26684ec006 (Wed, 19 Sep 2018 10:20:02 -0400):

Emilio Lahr-Vivaz does this cover some of what needs to be addressed with FSDS compactions? Or should we enter an additional ticket?

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Wed, 19 Sep 2018 10:28:59 -0400):

this is different I think, but might be fixed by the changing the metadata format

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Thu, 13 Dec 2018 11:38:06 -0500):

This is still a potential issue - given that we can't atomically modify/delete multiple files at once, I'm not sure of a good solution.

Start by locating the FSDS compaction command and the in-memory list of compacted files marked for deletion. Trace how that list and metadata are handled when the driver program is killed; done means compaction can recover without leaving data or metadata in an intermediate state.