Using hundreds of CAR files as a single block store is slow
- Dominant language
- Rust
- Stars
- 697
- Forks
- 200
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 65
Description
**Issue summary**
Multiple CAR files can be opened and joined into a single block store. This is convenient when using diff snapshots containing only the unique data generated over a range of epochs.
However, looking up a value in a `ManyCar` block store queries each CAR file. So if there are hundreds of CAR files, one lookup may query hundreds of files, one by one.
Steps to reproduce:
1. `cd calibnet_diff`
2. `aws --endpoint https://2238a825c5aca59233eab1f221f7aefb.r2.cloudflarestorage.com/ s3 cp "s3://forest-archive/calibnet/lite/forest_snapshot_calibnet_2022-11-01_height_0.forest.car.zst" .`
3. `aws --endpoint https://2238a825c5aca59233eab1f221f7aefb.r2.cloudflarestorage.com/ s3 cp "s3://forest-archive/calibnet/diff/" . --recursive`
4. `forest-cli snapshot validate --check-links 780000 --check-stateroots 780000 forest_snapshot_calibnet_2022-11-01_height_0.forest.car.zst forest_diff_calibnet_height_*`
One possible solution would be to order the CAR files according to when they were last used. Getting a successful hit from a CAR file should move it to the front of the list, ensuring active CAR files are queried first. This will improve performance if nearby epochs are accessed together rather than randomly.
Acceptance criteria:
- [ ] Traversing 2000 epochs with 260 CAR files should be, at most, 10% slower than traversing with 10 CAR files.
**Other information and links**
Contributor guide
Assessment
This issue has not been assessed yet.