ChainSafe / ChainSafe/forest

Automatically upload diff and lite snapshots to CloudFlare

Open
#3,577 0 comments 0 reactions 0 assignees View on GitHub
Priority: 2 - High Ready Type: Feature
Dominant language
Rust
Stars
697
Forks
200
Avg merge
1d 5h
Merged PRs (30d)
65

Description

**Issue summary**

Forest stores diff snapshots (covering 3k epochs) and lite snapshots (every 30k epochs) in CloudFlare. This archive goes all the way back to genesis, and it should be regularly updated with new snapshots. More information about the blockchain graph can be found here: https://github.com/ChainSafe/forest/blob/main/documentation/src/developer_documentation/filecoin_archive.md

Once per day, a new diff snapshot should be uploaded. This snapshot only contains the new key-value pairs that were written in the last 3k epochs. Once per deca-day (every 10 days), a new lite snapshot should be uploaded. Its epoch should cleanly divide 30,000.

It is unclear how to schedule this work best, and it is up to the issue owner to find an appropriate solution. One option would be a cron workflow that executes a script on `fuzzy`. Another option would be to spawn a DigitalOcean droplet.

Tasks:
- [ ] Write Rust script that checks for missing diff/lite snapshots.
- [ ] If a snapshot is missing, sync with Forest and export/upload the snapshot.
- [ ] Report failures/successes to Slack.
- [ ] Figure out a sensible way of triggering the script at least once per day.

**Other information and links**

CloudFlare bucket: https://dash.cloudflare.com/2238a825c5aca59233eab1f221f7aefb/r2/default/buckets/forest-archive

Sample lite snapshot names:
```
forest_snapshot_mainnet_2023-09-13_height_3210000.forest.car.zst
forest_snapshot_mainnet_2023-09-24_height_3240000.forest.car.zst
forest_snapshot_mainnet_2023-09-24_height_3270000.forest.car.zst
```

Sample diff snapshot names:
```
forest_diff_mainnet_2021-03-10_height_570000+3000.forest.car.zst
forest_diff_mainnet_2021-03-11_height_573000+3000.forest.car.zst
forest_diff_mainnet_2021-03-12_height_576000+3000.forest.car.zst
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.