apache / apache/incubator-pegasus
Feature: decrease bulk load ingestion executing time
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Feature Request
Pegasus has supported bulk load function, rely on [Rocksdb ingestion](https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files). When ingesting data, Rocksdb will reject write request, so Pegasus will also reject user write requests during bulk load ingesting stage. To decrease this time, Pegasus should modify Rocksdb related code and pay attention to Rocksdb recent releases.
Contributor guide
Research direction
Start by reviewing Pegasus's existing bulk-load ingestion path and the linked RocksDB SST ingestion documentation, then compare the relevant behavior in recent RocksDB releases. Measure when user writes are rejected during ingestion and define a design that reduces that interval; done requires a verified reduction without breaking bulk-load behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100