Concurrent major compactions could possibly cause metadata inconsistencies
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
When a major compaction finishes [this function](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L451) does the following.
1. Updates the tablets in memory data structures
2. Updates the tablets files in the metadata table
Once step 1 is complete above, those files are available for a subsequent compaction. That could possibly lead to the following race condition with concurrent compactions on the same tablet.
1. Compaction 1 starts compacting files [F1,F2] into file C3
2. Compaction 1 removes [F1,F2] and adds C3 to the tablets in memory file set
3. Compaction 2 starts compacting file [F0,C3] into file C4
4. Compaction 2 removes [F0,C3] and adds C4 to the tablets in memory file set
5. Compaction 2 removes [F0,C3] and adds C4 to the tablets row in the metadata table
6. Compaction 1 removes [F1,F2] and adds C3 to the tablets row in the metadata table
The above race condition adds the file C3 back to the metadata table when it should not. This could bring back deleted data and would cause the tablet to report memory and metadata as inconsistent.
This race condition probably only exists between concurrent major compactions. For [bulk import](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L214) and [minor compaction](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L304) this race condition is probably not possible because those functions update the metadata table and then the in memory set. Its updating the in memory set first that makes it available for compaction and leads to the possible race condition.
This was likely not an issue in Accumulo 1.X because tablets could only have a single compaction running at a time. With the introduction of concurrent tablet compactions in 2.x it seems like this race condition could happen, but not sure if there is something in the code outside of DataFileManager that accidentally prevents the race condition from happening.
**Expected behavior**
Race condition is not possible
Contributor guide
Research direction
Start in server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java at the major-compaction path linked in the issue, then compare it with the bulk-import and minor-compaction paths. Trace concurrent major compactions through the in-memory file set and metadata table, and establish that the reported race cannot leave metadata and memory inconsistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100