apple / apple/foundationdb

Add support for bulk data loading that bypasses transaction logs

Open
#1,002 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

This feature will improve write bandwidth when initially loading data into a database. During the load, the range will not be readable. The design needs to consider how to get back to a consistent view of the world if a client dies during loading.

Proposed design:
* verify the range is empty, lock reads to the range, writes to the range that come from the tlogs are discarded
* disable data movement for the range
* directly send mutations to the server responsible for the range
* if one of the servers dies while loading mutations, clear the loaded data from the other servers and unlock the range
* unlocked the range
* re-enable data movement for the range

Contributor guide

Open the contributing guide

Research direction

No files or tests are identified. Start by reviewing the proposed range-locking, transaction-log, direct-mutation, failure-recovery, and data-movement steps in the issue, then locate the corresponding implementation entry points; done would require a complete bulk-loading design with consistent recovery when a server dies.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.