Garbage Collector removed referenced files

A significant number of Imported files were garbage collected despite the presence of "file" entries referencing them. This impacted only the file references within one accumulo.metadata tablet which supported many tablets for a table.

The tablet server hosting the accumulo.metadata tablet died, the tablet was reassigned, and the WALs were recovered. See the logs summary below.

16:04:50 - tserver imports one of the files that will later go missing
17:28:59 - GC recognizes candidate file is in use
17:41:25 - GC gc.SimpleGarbageCollector collect cycle statistics reported
17:42:40 - GC gc.GarbageCollectWriteAheadLogs reports statistics
17:44:49 - GC Thread gc stuck on IO to accumulo master for at least 2 minutes
17:46:26 - GC Thread gc no longer stuck
17:56:06 - datanode dies on tserver hosting accumulo.metadata
...
Tserver reports many errors related to HDFS access, including WAL delete failures
...
17:57:14 - datanode startup
17:57:44 - tserver dies

17:58:45 - master.Master reports tablet assigned to dead tserver
17:58:49 - master.Master reports old location state
17:59:20 - master.Master reports old location state
17:59:20 - master.Master reports assigning tablet
17:59:27 - master.Master reports new location state
17:59:28 - master.Master reports new location state
17:59:29 - master.Master reports new location state
17:59:26 - tserver: Loading accumulo.metadata tablet
17:59:31 - tserver: got assignment from master for tablet, loading and verifying extent
17:59:32 - Starting WAL recovery occurs accumulo.metadata tablet
17:59:32 - WAL recovery complete for accumulo.metadata tablet
17:59:34 - master.EventCoordinator reports accumulo.metadata tablet loaded

18:06:09 - GC deletes file that was previously recognized as in use

18:10:29 - gc.SimpleGarbageCollector collect cycle statistics reported
18:12:00 - Errors begin for one of the missing files

This impacted a 1.8.1 fork that contained the 1.9.1 dataloss bug fixes.

Start with the incident timeline and the named gc.SimpleGarbageCollector and gc.GarbageCollectWriteAheadLogs components, then trace how accumulo.metadata tablet recovery and WAL processing affect file references. Done means identifying why a referenced imported file became a deletion candidate and establishing a fix or regression coverage that prevents deletion while the reference remains present.

Written by the indexing model from the issue text.