evstack / evstack/apex

Rate limiting and circuit breaker for transaction submission

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add backpressure mechanisms to the transaction submission path to handle DA layer degradation gracefully.

Parent: #4

## Context

celestia-node has zero rate limiting or backpressure on submission — just a mutex serializing broadcasts. Under load or during network issues, this leads to unbounded retries and resource exhaustion.

## Requirements

### Concurrency limiter
- Semaphore bounding max in-flight transactions (configurable, default: 4)
- Callers block when limit is reached
- Works with multi-account (#5): limit is global across all accounts

### Exponential backoff
- On repeated submission failures, increase delay between attempts
- Base: 1s, max: 30s, jitter: +/- 25%
- Reset backoff on successful submission

### Circuit breaker
- Track failure rate over a sliding window (e.g., last 20 submissions)
- States: closed (normal) → open (failing, reject immediately) → half-open (probe)
- Open threshold: >50% failure rate over window
- Half-open: allow 1 submission, if success → closed, if fail → open
- When open: return immediate error with "circuit open" context, don't waste gas

### Metrics (#7)
- `apex_submission_inflight` (gauge)
- `apex_submission_circuit_state` (gauge — 0=closed, 1=half-open, 2=open)
- `apex_submission_backoff_seconds` (gauge)

```toml
[submission.rate_limit]
max_inflight = 4
backoff_base = "1s"
backoff_max = "30s"
circuit_window = 20
circuit_open_threshold = 0.5
```

## References

- Sony's gobreaker or similar for circuit breaker pattern

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.