erigontech / erigontech/silkworm

Snapshot bundle cleanup across CAPI

Open
#2,678 0 comments 0 reactions 0 assignees View on GitHub
c api erigon3 snapshots
Dominant language
C++
Stars
318
Forks
84
PR merge metrics
No merged PRs in 30d

Description

Erigon has a reference counter attached to snapshot files in use by AggregatorRoTx. When AggregatorRoTx is closed (using Close()) it triggers a scheduled cleanup of files that have been merged into bigger files (closeFilesAndRemove). If AggregatorRoTx is not opened, refcount is zero and closeFilesAndRemove happens immediately after merging (deleteMergeFile).

Currently this is not coordinated with silkworm. While using snapshot files managed by erigon, silkworm is not beginning an AggregatorRoTx. If the files are merged, they will be unmapped, closed and deleted (mmap.Munmap). At this point SnapshotRepository will contain dangling references that will lead to errors or crashes.

We need to care about the following:
1. If snapshot files are currently in use by a datastore query, the usage refcount must be incremented on the Erigon side to prevent cleanup. When snapshot files are no longer in use - decrement refcount.
2. If files are cleaned up by Erigon, they must be removed from SnapshotRepository.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.