FilOzone / FilOzone/filecoin-pay

perf: terminated rail settlement advances ~230 epochs per 7 Ggas call on data sets with large piece histories - endEpoch is unreachable in practice

Open
#298 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Solidity
Stars
8
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Three terminated PDP rails on mainnet provider 1 (`ezpdpz-main`) have been settling twice a day since at least 17 Aug, advancing a fixed couple of hundred epochs each time.

Figures as at the 2026-08-25 03:35 UTC settle round:

| rail | data set | piece adds | settledUpTo | endEpoch | epochs left | epochs per call | gas per call (15 calls) |
|---|---|---|---|---|---|---|---|
| 91 | 77 | 1,004,049 | 5,722,823 | 6,197,243 | 474,420 | 231 | 5.3-8.8 Ggas, avg 6.9 |
| 76 | 64 | 195,610 | 5,724,469 | 6,348,463 | 623,994 | 285 | 6.0-6.6 Ggas, avg 6.2 |
| 2224 | 1245 | 435,851 | 6,098,142 | 6,241,800 | 143,658 | 140 | 5.4-9.5 Ggas, avg 7.4 |

Rail 2484 on the same node (data set 1424, 4,610 piece adds) gains 452 to 1,293 epochs per call, avg around 750, at avg 6.5 Ggas.

At these rates rail 91 needs about 1,030 days, rail 76 about 1,095, rail 2224 about 515. Extrapolated from 15 settle rounds each, so treat as an order of magnitude.

## Effect

The settle calls share the SP's single sender wallet with proving and `addPieces`. Send to confirm times from my node for the 2026-08-23 15:35 UTC round, 19 settle messages sent between 15:34:42 and 15:35:32:

```
nonce sent confirmed latency reason
1397248 15:34:42 15:35:00 18s settleRail
1397254 15:34:53 15:38:00 187s settleRail
1397263 15:35:17 15:45:00 582s settleRail
1397265 15:35:22 15:45:30 608s pdp-addpieces
1397267 15:35:30 15:45:30 600s settleRail
1397274 15:50:38 15:51:00 21s pdp-addpieces
```

The `addPieces` at nonce 1397265 sat behind 17 settle messages and took 608 seconds. The next one, into a clear lane 15 minutes later, took 21. Dealbot's on chain confirmation window is 300s, so that deal was scored failed while the data was stored and the transaction landed 5 minutes after the timeout.

Six deals were scored failed this way, all with the same error, on the 03:40 and 15:40 rounds:

```
08-21 15:40:03 08-22 03:40:17 08-22 15:40:05
08-23 03:40:33 08-23 15:40:22 08-24 15:41:34
Failed to commit on primary provider 1 - data is stored but not on-chain
Request timed out after 300000ms
```

Proving is unaffected, 0 faults across 945 proving periods in the week of 17 Aug.

Not chain congestion. Base fee ran 100 to 113 attoFIL across the window and tipsets were 0.3% to 24% full. Not send side either, all 19 messages went out inside 50 seconds.

The only lever I could find is `settleTerminatedRailWithoutValidation`, which is client-only, so there is nothing I can do about this from the SP side.

**Related**

FilOzone/filecoin-services#546, #228, filecoin-project/curio#1313.

Happy to pull more off the node if any of it helps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the client-only settleTerminatedRailWithoutValidation path and compare it with the related issues FilOzone/filecoin-services#546, #228, and filecoin-project/curio#1313. Use the reported rail histories, settlement rates, gas usage, and sender-wallet delays as reproduction evidence. Done should include an identified cause and a validated change that prevents terminated-rail settlement from delaying addPieces transactions.

Written by the indexing model from the issue text.

Assessment

Tech stack
solidity
Domain
blockchain
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.