Index should represent the log more accurately when DELETE and PUT occur in the same segment
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.8k
- Forks
- 298
- Avg merge
- 21h 43m
- Merged PRs (30d)
- 9
Description
Currently, if a DELETE for a blob occurs before the segment in which the PUT entry for the blob exists rolls over, the DELETE overwrites the PUT. This might result in an inaccurate representation of the log.
Particularly, during crashes, consider the following scenario
(entries are of the form ID:TYPE:START_OFFSET-END_OFFSET)
This is the order of operations
A:PUT:0-100
B:PUT:101-200
A:DELETE:201-250
These are the entries in the index segment
A:DELETE:201-250
B:PUT:101-200
If safeEndPoint < 250, then B:PUT:101-200 will be written but not A:DELETE:201-250. If the process were to crash at this point, the index end offset would be 200 and the span 0-100 would not be represented in the index.
This results in 1) Missing extents of the log in the index 2) recovery creating "orphaned" delete entries that have no reference to the PUT record even though it exists. We need to work on a holistic solution in the index segment that can accurately represent both the PUT and DELETE without overwrites. This way, we will not lose information.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the index segment and recovery behavior described in the issue. Reproduce the A:PUT, B:PUT, A:DELETE sequence and examine how safeEndPoint affects persisted entries after a crash. Done means the index preserves both PUT and DELETE information without losing log extents or creating orphaned delete entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100