Add support for bulk data loading that bypasses transaction logs
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
This feature will improve write bandwidth when initially loading data into a database. During the load, the range will not be readable. The design needs to consider how to get back to a consistent view of the world if a client dies during loading.
Proposed design:
* verify the range is empty, lock reads to the range, writes to the range that come from the tlogs are discarded
* disable data movement for the range
* directly send mutations to the server responsible for the range
* if one of the servers dies while loading mutations, clear the loaded data from the other servers and unlock the range
* unlocked the range
* re-enable data movement for the range
Contributor guide
Research direction
No files or tests are identified. Start by reviewing the proposed range-locking, transaction-log, direct-mutation, failure-recovery, and data-movement steps in the issue, then locate the corresponding implementation entry points; done would require a complete bulk-loading design with consistent recovery when a server dies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100