flashbots / flashbots/hindsight
batch fetch/export process for `export` command
Open
- Dominant language
- Rust
- Stars
- 130
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
`hindsight export` currently loads ALL entries from the source db into memory before dumping them into the destination.
This is reeeal slow on a large dataset (1+ months).
proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save
- file export will need modification to append to the file, also thread-safe lock on the file ofc
- db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.