filecoin-project / filecoin-project/lotus

GPU preemption failure

Open
#9,727 0 comments 0 reactions 0 assignees View on GitHub
area/miner-maintenance kind/bug LM-tech-debt
Dominant language
Go
Stars
3k
Forks
1.4k
Avg merge
3d 2h
Merged PRs (30d)
28

Description

### Checklist

- [X] This is **not** a security-related bug/issue. If it is, please follow please follow the [security policy](https://github.com/filecoin-project/lotus/security/policy).
- [X] This is **not** a question or a support request. If you have any lotus related questions, please ask in the [lotus forum](https://github.com/filecoin-project/lotus/discussions).
- [X] This is **not** a new feature request. If it is, please file a [feature request](https://github.com/filecoin-project/lotus/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Ffeature&template=feature_request.yml) instead.
- [X] This is **not** an enhancement request. If it is, please file a [improvement suggestion](https://github.com/filecoin-project/lotus/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Fenhancement&template=enhancement.yml) instead.
- [X] I **have** searched on the [issue tracker](https://github.com/filecoin-project/lotus/issues) and the [lotus forum](https://github.com/filecoin-project/lotus/discussions), and there is no existing related issue or discussion.
- [X] I am running the [`Latest release`](https://github.com/filecoin-project/lotus/releases), or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
- [ ] I did not make any code changes to lotus.

### Lotus component

- [ ] lotus daemon - chain sync
- [ ] lotus miner - mining and block production
- [ ] lotus miner/worker - sealing
- [ ] lotus miner - proving(WindowPoSt)
- [ ] lotus miner/market - storage deal
- [ ] lotus miner/market - retrieval deal
- [ ] lotus miner/market - data transfer
- [ ] lotus client
- [ ] lotus JSON-RPC API
- [ ] lotus message management (mpool)
- [ ] Other

### Lotus Version

```text
lotus version 1.18.0+mainnet+git.bd10bdf99
```

### Describe the Bug

When the wdpost calculation and winningpost calculation occur at the same time, although the priority of winningpost is true and that of wdpost is false, winningpost still fails to preempt the GPU,and then winningpost computing timeout

### Logging Information

```text
2022-11-24T23:07:17.607 INFO storage_proofs_core::compound_proof > snark_proof:start
2022-11-24T23:07:17.750 INFO bellperson::groth16::prover > Bellperson 0.22.0 is being used!
2022-11-24T23:07:40.287Z INFO miner miner/miner.go:548 completed mineOne {"tookMilliseconds": 12, "forRound": 2367496, "baseEpoch": 2367495, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2463341, "lookbackEpochs": 900, "networkPowerAtLookback": "21798573920164216832", "minerPowerAtLookback": "11159012229775360", "isEligible": true, "isWinner": false, "error": null}
2022-11-24T23:07:55.019 INFO bellperson::groth16::prover > synthesis time: 37.268746346s
2022-11-24T23:07:55.019 INFO bellperson::groth16::prover > starting proof timer
2022-11-24T23:07:59.294 INFO bellperson::gpu::locks > GPU is available for FFT!
2022-11-24T23:07:59.295 INFO ec_gpu_gen::program > Using kernel on CUDA.
2022-11-24T23:07:59.317 INFO ec_gpu_gen::fft > FFT: 1 working device(s) selected.
2022-11-24T23:07:59.318 INFO ec_gpu_gen::fft > FFT: Device 0: GeForce RTX 3090
2022-11-24T23:07:59.318 INFO bellperson::gpu::locks > GPU FFT kernel instantiated!
2022-11-24T23:08:10.074Z INFO miner miner/miner.go:548 completed mineOne {"tookMilliseconds": 51, "forRound": 2367497, "baseEpoch": 2367495, "baseDeltaSeconds": 40, "nullRounds": 1, "lateStart": false, "beaconEpoch": 2463342, "lookbackEpochs": 900, "networkPowerAtLookback": "21798574221660848128", "minerPowerAtLookback": "11159012229775360", "isEligible": true, "isWinner": false, "error": null}
2022-11-24T23:08:17.474Z ERROR storagemarket_impl impl/provider.go:205 failed to connect index provider host with the full node: failed to call NetProtectAdd on the full node, err: missing permission to invoke 'NetProtectAdd' (need 'admin')
2022-11-24T23:08:27.245 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2022-11-24T23:08:27.245 INFO bellperson::gpu::multiexp > Multiexp: CPU utilization: 0.
2022-11-24T23:08:27.246 INFO ec_gpu_gen::program > Using kernel on CUDA.
2022-11-24T23:08:27.248 INFO ec_gpu_gen::multiexp > Multiexp: 1 working device(s) selected.
2022-11-24T23:08:27.248 INFO ec_gpu_gen::multiexp > Multiexp: Device 0: GeForce RTX 3090 (Chunk-size: 18061702)
2022-11-24T23:08:27.248 INFO bellperson::gpu::locks > GPU Multiexp kernel instantiated!
2022-11-24T23:08:40.246Z INFO miner miner/miner.go:548 completed mineOne {"tookMilliseconds": 17, "forRound": 2367498, "baseEpoch": 2367497, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2463343, "lookbackEpochs": 900, "networkPowerAtLookback": "21798572294495338496", "minerPowerAtLookback": "11159012229775360", "isEligible": true, "isWinner": false, "error": null}
2022-11-24T23:09:10.044Z INFO miner miner/miner.go:590 round winner, will mine new block, for {"height": "2367499"}
2022-11-24T23:09:10.045Z INFO storageminer storage/winning_prover.go:70 Computing WinningPoSt ;[{SealProof:9 SectorNumber:152313 SectorKey: SealedCID:bagboea4b5abcadqi47tmsbyg24t463o4u2nkb5dbhzn24ndpv7mb7ywjosqc4e27}]; [114 34 174 18 207 210 176 171 229 24 20 68 99 184 137 67 14 147 227 93 13 156 156 207 168 200 156 1 88 161 142 198]
2022-11-24T23:09:10.045Z INFO advmgr sealer/manager_post.go:23 GenerateWinningPoSt run at lotus-miner
2022-11-24T23:09:10.054 INFO filecoin_proofs::api::winning_post > generate_winning_post:start
2022-11-24T23:09:10.065 INFO filecoin_proofs::caches > trying parameters memory cache for: WINNING_POST[68719476736]
2022-11-24T23:09:10.065 INFO filecoin_proofs::caches > found params in memory cache for WINNING_POST[68719476736]
2022-11-24T23:09:10.191 INFO storage_proofs_core::compound_proof > vanilla_proofs:start
2022-11-24T23:09:10.488 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
2022-11-24T23:09:10.493 INFO storage_proofs_core::compound_proof > snark_proof:start
2022-11-24T23:09:10.494 INFO bellperson::groth16::prover > Bellperson 0.22.0 is being used!
2022-11-24T23:09:10.594 INFO bellperson::groth16::prover > synthesis time: 100.372952ms
2022-11-24T23:09:10.594 INFO bellperson::groth16::prover > starting proof timer
2022-11-24T23:09:10.610 INFO bellperson::gpu::locks > GPU is available for FFT!
2022-11-24T23:09:10.610 INFO ec_gpu_gen::program > Using kernel on CUDA.
2022-11-24T23:09:11.044 INFO ec_gpu_gen::fft > FFT: 1 working device(s) selected.
2022-11-24T23:09:11.044 INFO ec_gpu_gen::fft > FFT: Device 0: GeForce RTX 3090
2022-11-24T23:09:11.044 INFO bellperson::gpu::locks > GPU FFT kernel instantiated!
2022-11-24T23:09:17.476Z ERROR storagemarket_impl impl/provider.go:205 failed to connect index provider host with the full node: failed to call NetProtectAdd on the full node, err: missing permission to invoke 'NetProtectAdd' (need 'admin')
2022-11-24T23:10:17.477Z ERROR storagemarket_impl impl/provider.go:205 failed to connect index provider host with the full node: failed to call NetProtectAdd on the full node, err: missing permission to invoke 'NetProtectAdd' (need 'admin')
2022-11-24T23:11:00.420 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2022-11-24T23:11:00.420 INFO bellperson::gpu::multiexp > Multiexp: CPU utilization: 0.
2022-11-24T23:11:00.420 INFO ec_gpu_gen::program > Using kernel on CUDA.
2022-11-24T23:11:00.421 INFO bellperson::groth16::prover > prover time: 185.401883711s
2022-11-24T23:11:01.763 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-11-24T23:11:01.763 INFO filecoin_proofs::api::window_post > generate_window_post:finish
2022-11-24T23:11:01.764 INFO ec_gpu_gen::multiexp > Multiexp: 1 working device(s) selected.
2022-11-24T23:11:01.764 INFO ec_gpu_gen::multiexp > Multiexp: Device 0: GeForce RTX 3090 (Chunk-size: 18061702)
2022-11-24T23:11:01.764 INFO bellperson::gpu::locks > GPU Multiexp kernel instantiated!
2022-11-24T23:11:02.383 INFO bellperson::groth16::prover > prover time: 111.788384309s
2022-11-24T23:11:02.387 INFO storage_proofs_core::compound_proof > snark_proof:finish
2022-11-24T23:11:02.387 INFO filecoin_proofs::api::winning_post > generate_winning_post:finish
2022-11-24T23:11:02.388Z INFO storageminer storage/winning_prover.go:77 GenerateWinningPoSt took 1m52.342815192s
2022-11-24T23:11:04.035Z INFO wdpost wdpost/wdpost_run.go:732 computing window post {"batch": 0, "elapsed": 301.29480264, "skip": 0, "err": null}
2022-11-24T23:11:04.047 INFO filecoin_proofs::api::window_post > verify_window_post:start
2022-11-24T23:11:04.047 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[68719476736]-verifying-key
2022-11-24T23:11:04.047 INFO filecoin_proofs::caches > found params in memory cache for WINDOW_POST[68719476736]-verifying-key
2022-11-24T23:11:04.081 INFO filecoin_proofs::api::window_post > verify_window_post:finish
2022-11-24T23:11:06.994Z INFO miner miner/miner.go:645 mined new block {"cid": "bafy2bzaceaukxlerk4p4rjtq6x6yfdn764vjmtbkmr2a7wunn3mnaqov2if2a", "height": 2367499, "miner": "f0502198", "parents": ["f0230861","f01886704","f01702940","f01171513","f01852363","f01680940","f089180","f01926802"], "parentTipset": "{bafy2bzaceduwoeqccvx34e6qosnrzjteg4qelshpxnnmdzqcl33axfjyng3qy,bafy2bzaceabyjgwvovms7vdayfuhcqfhyv4ufbvgtnzj66dt7ksdyakzky7ms,bafy2bzacebnwfa24777iglbvuebudz3ockjqvu3adoqmwqd52dtcbgevmeaqo,bafy2bzacea4b5n6olk6khgla6ntji3oacebljbfywakph6jv2buev2ygbterm,bafy2bzacebl76ealt5wv2nzmxrzbb3saoynuvc3brfwuldltbdiyb2kwu42mi,bafy2bzacec3qgh53icxqezayboosxobtoznilaemn47qqqb5hre7corkqyns6,bafy2bzacedyn2wahnuyaaahtv5nmoub2orhwsfob7ovk5gxn6lqfkcynnmby4,bafy2bzaceasooknnkcphdfsuigvbeet3dzrzhmmcx4v6g335liahl3su73ywm}", "took": 116.967937239}
2022-11-24T23:11:06.994Z WARN miner miner/miner.go:647 CAUTION: block production took longer than the block delay. Your computer may not be fast enough to keep up {"tPowercheck ": 0.016822184, "tTicket ": 0.0015335, "tSeed ": 0.00000244, "tProof ": 112.343196209, "tPending ": 4.507330629, "tCreateBlock ": 0.099052277}
```

### Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
...

Contributor guide

Open the contributing guide

Research direction

Start by tracing the GPU-related work through storage/winning_prover.go and wdpost/wdpost_run.go, using the timing in the supplied logs to compare WinningPoSt and WindowPoSt execution. Check how the miner path in miner/miner.go coordinates these operations. Done means concurrent calculations honor WinningPoSt priority, preemption occurs as expected, and WinningPoSt avoids the reported timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.