erigontech / erigontech/erigon

db/downloader: batch-owned download stats instead of one global sample

Open
#23,415 0 comments 0 reactions 0 assignees View on GitHub
ErigonDB
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

`Downloader.lastStats` is a single slot holding the latest `AggStats`. It is written by the per-batch logging loop (`logDownload` → `storeStats`) and read by `Completed()`, and the samples carry no batch identity.

## Why it matters

`storeStats` holds `BytesCompleted` at the previous sample's level to keep reported progress monotonic, and that max crosses batches, while `BytesTotal` belongs to whichever batch wrote last. Two overlapping batches therefore produce `done > total`, and the snapshot-download progress reporter freezes on a ratio above 1.

#22716 mitigated this by serializing batches: `DownloadSnapshots` holds `downloadBatchLock` for the whole call. That enforces the "one download session at a time" contract the function already documented, but it is coarse — `Download` is a public RPC, and a retry after a client-side transport error can produce a second call, which now queues for the entire download instead of running.

## Proposal

Give each download batch its own counters, have `Completed()` report the active batch's numbers (or report nothing while more than one batch is active), then remove `downloadBatchLock` so concurrent batches are possible again without blending samples.

## Acceptance

Two overlapping `Download` calls never yield `done > total` from `Completed()`, and neither blocks the other.

Follow-up to #22716.

Contributor guide

Open the contributing guide

Research direction

Trace Downloader.lastStats through logDownload, storeStats, and Completed(), then inspect DownloadSnapshots and downloadBatchLock. Define per-batch counters so overlapping Download calls can proceed without mixing samples; done means concurrent calls never make Completed() report done greater than total and neither call blocks the other.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.