ChainSafe / ChainSafe/lodestar

Add retry mechanism for payload envelope DB persistence

Open
#9,072 0 comments 0 reactions 0 assignees View on GitHub
spec-gloas
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

## Context

In `persistPayloadEnvelopeInput()`, a DB write failure causes the in-memory `PayloadEnvelopeInput` to be pruned via `.finally()`, losing data from both DB and memory with no recovery path. This is the same pattern as `persistBlockInput()`.

### Current behavior

```typescript
await writePayloadEnvelopeInputToDb.call(this, payloadInput)
.catch((e) => {
this.logger.error("Error persisting payload envelope in hot db", {...}, e);
})
.finally(() => {
this.seenPayloadEnvelopeInput.prune(payloadInput.blockRootHex);
// ...
});
```

On a transient DB error (disk full, I/O timeout), `.catch()` logs the error but `.finally()` still prunes the in-memory data. The envelope is now gone from both DB and memory — irrecoverable.

## Proposed fix

1. Only prune on success (move prune from `.finally()` to `.then()`)
2. Add a retry mechanism for transient failures (exponential backoff, bounded retries)
3. Apply the same fix to `persistBlockInput()` which has the identical pattern

## References

- PR #8962 — `writePayloadEnvelopeInputToDb.ts` ([review thread](https://github.com/ChainSafe/lodestar/pull/8962#discussion_r2966245758))
- Same pattern exists in `persistBlockInput()` (noted by @twoeths)

/cc @nflaig @twoeths

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.