apache / apache/accumulo

Concurrent major compactions could possibly cause metadata inconsistencies

Open
#3,740 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Describe the bug**
When a major compaction finishes [this function](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L451) does the following.

1. Updates the tablets in memory data structures
2. Updates the tablets files in the metadata table

Once step 1 is complete above, those files are available for a subsequent compaction. That could possibly lead to the following race condition with concurrent compactions on the same tablet.

1. Compaction 1 starts compacting files [F1,F2] into file C3
2. Compaction 1 removes [F1,F2] and adds C3 to the tablets in memory file set
3. Compaction 2 starts compacting file [F0,C3] into file C4
4. Compaction 2 removes [F0,C3] and adds C4 to the tablets in memory file set
5. Compaction 2 removes [F0,C3] and adds C4 to the tablets row in the metadata table
6. Compaction 1 removes [F1,F2] and adds C3 to the tablets row in the metadata table

The above race condition adds the file C3 back to the metadata table when it should not. This could bring back deleted data and would cause the tablet to report memory and metadata as inconsistent.

This race condition probably only exists between concurrent major compactions. For [bulk import](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L214) and [minor compaction](https://github.com/apache/accumulo/blob/b1b2557f949e9212a1b1ca9b65f2d66c01a69edb/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L304) this race condition is probably not possible because those functions update the metadata table and then the in memory set. Its updating the in memory set first that makes it available for compaction and leads to the possible race condition.

This was likely not an issue in Accumulo 1.X because tablets could only have a single compaction running at a time. With the introduction of concurrent tablet compactions in 2.x it seems like this race condition could happen, but not sure if there is something in the code outside of DataFileManager that accidentally prevents the race condition from happening.

**Expected behavior**
Race condition is not possible

Contributor guide

Open the contributing guide

Research direction

Start in server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java at the major-compaction path linked in the issue, then compare it with the bulk-import and minor-compaction paths. Trace concurrent major compactions through the in-memory file set and metadata table, and establish that the reported race cannot leave metadata and memory inconsistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.