flashbots / flashbots/hindsight

batch fetch/export process for `export` command

Open
#19 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
130
Forks
19
PR merge metrics
No merged PRs in 30d

Description

`hindsight export` currently loads ALL entries from the source db into memory before dumping them into the destination.

This is reeeal slow on a large dataset (1+ months).

proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save
- file export will need modification to append to the file, also thread-safe lock on the file ofc
- db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.