ChainSafe / ChainSafe/forest

Query CAR database according to epoch

Open
#3,555 1 comment 0 reactions 0 assignees View on GitHub
Performance Priority: 2 - High Ready Type: Enhancement
Dominant language
Rust
Stars
697
Forks
200
Avg merge
1d 5h
Merged PRs (30d)
65

Description

**Issue summary**

There are two kinds of data stores in Forest: Read-only CAR files and writeable ParityDB databases. Since all values are uniquely determined by their key, it does not matter for correctness which data store we query first. It does matter for performance, though, and we want to query as few data stores as possible.

We first query the CAR data stores in the order they were added. If a key isn't present in the CAR data stores, we then query the ParityDB database. However, in regular operation, we're significantly more likely to query _new_ data than _old_ data. As such, querying the data stores with the latest data first is a better option. The ParityDB database contains current data and is, therefore, the newest and should be queried first. The CAR data stores should be sorted from highest to lowest by epoch.

- [x] Query ParityDB first.
- [ ] Sort CAR data stores by the epoch of the heaviest tipset.
- [ ] (Optional) Trim unusable CAR data stores if we know we'll never access their data. This happens if we're evaluating tipset 1000, and we have CAR stores for epochs 0-1500 and 1500-3000. The second CAR store will not be used for evaluating this tipset and can be removed from the list.

**Other information and links**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.