NVIDIA / NVIDIA/aistore

Concurrent DELETE download jobs crashes the target

Open
#367 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
1.9k
Forks
285
Avg merge
3d 17h
Merged PRs (30d)
10

Description

Is there an existing issue for this?
  • I have searched the existing issues
Describe the bug

To reproduce, generate some download job history, and run concurrent requests to delete some of them (through the DELETE /v1/download/remove endpoint)

Expected Behavior
  • Ideally requests are handled concurrently
  • Or less ideally some requests are failed but retryable, without crashing the server application
Current Behavior

Target pod crashes with

fatal error: concurrent map writes

goroutine 2205552 [running]:
internal/runtime/maps.fatal({0x34d84f3?, 0x2f73540?})
	runtime/panic.go:1181 +0x18
internal/runtime/maps.(*Map).Delete(0x19d6f4088390, 0x2f73540, 0x19d6f3d19470)
	internal/runtime/maps/map.go:682 +0x125
github.com/NVIDIA/aistore/ext/dload.(*infoStore).delJob(...)
	github.com/NVIDIA/aistore/ext/dload/infostore.go:126
github.com/NVIDIA/aistore/ext/dload.(*dispatcher).handleRemove(0x36f4da0?, 0x19d6f3d19688)
	github.com/NVIDIA/aistore/ext/dload/dispatcher.go:383 +0xfc
github.com/NVIDIA/aistore/ext/dload.(*dispatcher).adminReq(0x19d6f4e05400, 0x19d6f3d19688)
	github.com/NVIDIA/aistore/ext/dload/dispatcher.go:365 +0x148
github.com/NVIDIA/aistore/ext/dload.(*Xact).RemoveJob(0x19d6f1fa2480, {0x19d6f6c008f0?, 0x34b03eb?})
	github.com/NVIDIA/aistore/ext/dload/xact.go:244 +0x85
github.com/NVIDIA/aistore/ais.(*target).downloadHandler(0x19d6f1fa2180, {0x36d5008, 0x19d6f3e064b0}, 0x19d6f3d1c640)
	github.com/NVIDIA/aistore/ais/tgtdl.go:178 +0xb1b
Steps To Reproduce

As mentioned above, happy to provide a more step by step script if needed.

Possible Solution

Seems like a mutex fix needed.

Additional Information/Context

No response

AIStore build/version

v5.0

Environment details (OS name and version, etc.)

Running in kubernetes with the recommended operator deployment charts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ext/dload/infostore.go at infoStore.delJob, then trace the DELETE /v1/download/remove path through dispatcher.go, xact.go, and ais/tgtdl.go. Reproduce concurrent deletion of download jobs and verify that requests no longer crash the target, while any rejected requests remain retryable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.