HathorNetwork / HathorNetwork/tx-mining-service
Add retry/backoff around block submission to avoid noise from transient fullnode failures
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add retry/backoff around block submission in `_push_block_job()` so that transient fullnode failures don't immediately surface as `Error when submitting block` log errors (which generate OpsGenie alert noise).
## Background
`_push_block_job()` in [`txstratum/manager.py`](https://github.com/HathorNetwork/tx-mining-service/blob/master/txstratum/manager.py) logs an error on the first `PushTxFailed`:
```python
try:
await self.backend.push_tx_or_block(job.get_data())
except PushTxFailed:
self.log.error("Error when submitting block", job=job)
```
`PushTxFailed` is raised by `python-hathorlib`'s `push_tx_or_block()` on a non-2xx fullnode response. In practice these are usually **transient** fullnode conditions — rate-limit (429), short out-of-sync windows (503/504), or a now-stale block — i.e. the same fullnode-dependency chain behind the wider tx-mining block-template alert family.
This was triaged in on-call ([on-call-incidents#260](https://github.com/HathorNetwork/on-call-incidents/issues/260)) and is currently silenced via a rate-limited log-manager whitelist (`5/1h`). The whitelist stops the noise but the underlying behavior (no retry) remains; a short fullnode blip still produces an error log on the first failure.
Related analysis: `ops-tools/docs/on-call/reports/20260514-open-alerts-review/30-group-3-tx-mining.md`. Related past incidents: on-call-incidents #1, #10, #42, #160, #203.
## Proposed change
- Add a bounded retry with backoff around `push_tx_or_block()` for block submission (a few attempts, short exponential backoff), only logging `error` once all retries are exhausted.
- Consider distinguishing retryable statuses (429/502/503/504, connection/DNS errors) from genuinely non-retryable rejections, and log the HTTP status / backend host in the failure message to aid future triage.
- Same reasoning likely applies to the other emitters in this chain (`update_block_template()` / block-template fetch), so consider a shared retry policy.
## Acceptance criteria
- Transient single-failure block submissions are retried and do not emit an `Error when submitting block` error log unless retries are exhausted.
- The error log, when it does fire, includes enough context (status code, backend) to triage.
## Notes
This complements (does not replace) the log-manager whitelist. Once retries land, the whitelist rate limit can be revisited/removed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in txstratum/manager.py at _push_block_job() and trace push_tx_or_block() and PushTxFailed handling. Define the bounded retry/backoff and retryable failure scope from the issue, then verify that transient failures avoid the error log and exhausted failures include status and backend context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100