Concurrent DELETE download jobs crashes the target
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 285
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 10
Description
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
To reproduce, generate some download job history, and run concurrent requests to delete some of them (through the DELETE /v1/download/remove endpoint)
Expected Behavior
- Ideally requests are handled concurrently
- Or less ideally some requests are failed but retryable, without crashing the server application
Current Behavior
Target pod crashes with
fatal error: concurrent map writes
goroutine 2205552 [running]:
internal/runtime/maps.fatal({0x34d84f3?, 0x2f73540?})
runtime/panic.go:1181 +0x18
internal/runtime/maps.(*Map).Delete(0x19d6f4088390, 0x2f73540, 0x19d6f3d19470)
internal/runtime/maps/map.go:682 +0x125
github.com/NVIDIA/aistore/ext/dload.(*infoStore).delJob(...)
github.com/NVIDIA/aistore/ext/dload/infostore.go:126
github.com/NVIDIA/aistore/ext/dload.(*dispatcher).handleRemove(0x36f4da0?, 0x19d6f3d19688)
github.com/NVIDIA/aistore/ext/dload/dispatcher.go:383 +0xfc
github.com/NVIDIA/aistore/ext/dload.(*dispatcher).adminReq(0x19d6f4e05400, 0x19d6f3d19688)
github.com/NVIDIA/aistore/ext/dload/dispatcher.go:365 +0x148
github.com/NVIDIA/aistore/ext/dload.(*Xact).RemoveJob(0x19d6f1fa2480, {0x19d6f6c008f0?, 0x34b03eb?})
github.com/NVIDIA/aistore/ext/dload/xact.go:244 +0x85
github.com/NVIDIA/aistore/ais.(*target).downloadHandler(0x19d6f1fa2180, {0x36d5008, 0x19d6f3e064b0}, 0x19d6f3d1c640)
github.com/NVIDIA/aistore/ais/tgtdl.go:178 +0xb1b
Steps To Reproduce
As mentioned above, happy to provide a more step by step script if needed.
Possible Solution
Seems like a mutex fix needed.
Additional Information/Context
No response
AIStore build/version
v5.0
Environment details (OS name and version, etc.)
Running in kubernetes with the recommended operator deployment charts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ext/dload/infostore.go at infoStore.delJob, then trace the DELETE /v1/download/remove path through dispatcher.go, xact.go, and ais/tgtdl.go. Reproduce concurrent deletion of download jobs and verify that requests no longer crash the target, while any rejected requests remain retryable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100