Slow-but-successful checks are recorded as permanent failures: track a degraded outcome class
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 9
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
## Problem
A check that completes successfully but slower than our window is recorded identically to one that never worked. Deal checks have a single hard cutoff (`dealJobTimeoutSeconds`, default 360s in [`config/constants.ts`](https://github.com/FilOzone/dealbot/blob/main/apps/backend/src/config/constants.ts#L15)). When the window expires, [`classifyFailureStatus`](https://github.com/FilOzone/dealbot/blob/main/apps/backend/src/metrics-prometheus/check-metric-labels.ts#L48) emits `failure.timedout` and the deal row is persisted with `status = failed` ([`deal.service.ts#L664`](https://github.com/FilOzone/dealbot/blob/main/apps/backend/src/deal/deal.service.ts#L664)). Nothing revisits that row when the addPieces message lands on chain later, so an eventually-successful deal stays a permanent failure in the DB and in every downstream metric.
The 2026-07-20/21 mainnet congestion made this concrete (internal discussion: [Slack thread](https://filecoinproject.slack.com/archives/C08TVNKJV7C/p1784556015711599)): elevated base fee pushed addPieces confirmations well past the deal window while the data itself stored and retrieved fine, and dealbot recorded deal failures. Those false negatives are indistinguishable from an SP that lost the data.
## What this tracks
1. An outcome class between success and failure: "worked, but outside our time requirement" (e.g. a `pending_confirm`-style state; naming open). Metrics and dashboards need to separate "SP broken" from "SP or chain slow".
2. Headroom between the optimal window and a longer functional-but-degraded window, instead of one binary cutoff. #540 documented how a single cutoff turns a continuous latency distribution into cliff-shaped failure rates.
3. Reconciling `failed` rows when the deal later confirms on chain (overlaps #465).
## Related
- #540 (silent `failure.timedout`, binary-cutoff artifact), #603 (slow RPC recorded as hard deal failures), #267 (timeout tuning), #465 (DB vs chainstate sync)
- filecoin-project/curio#1354 (zero-headroom message caps stranding under high base fee), FilOzone/infra#325
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the timeout configuration in apps/backend/src/config/constants.ts, classifyFailureStatus in apps/backend/src/metrics-prometheus/check-metric-labels.ts, and persistence around deal.service.ts:664. Review related issues #465, #540, and #603 to understand reconciliation and cutoff behavior. Done means delayed on-chain confirmations have a distinct degraded outcome, are reconciled in storage, and are separated in metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, typescript
- Domain
- backend, databases, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100