nspcc-dev / nspcc-dev/neofs-node
shard: Blobstor `file exists` errors move shard into degraded mode
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
looks like a bug but needs to be investigated. When local storage is loaded by objects with existing files in blobstor, shard shards switches to degraded mode fast and stops storage service
warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 1, "error": "could not put object to BLOB storage: file exists"}
warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 2, "error": "could not put object to BLOB storage: file exists"}
warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 3, "error": "could not put object to BLOB storage: file exists"}
warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 4, "error": "could not put object to BLOB storage: file exists"}
...
info shard/mode.go:28 setting shard mode {"shard_id": "ERHCRoaBixN42afWgnJrzA", "old_mode": "READ_WRITE", "new_mode": "DEGRADED_READ_ONLY"}
info shard/mode.go:69 shard mode set successfully {"shard_id": "ERHCRoaBixN42afWgnJrzA", "mode": "DEGRADED_READ_ONLY"}
info engine/engine.go:105 shard is moved in degraded mode due to error threshold {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 30}
...
warn engine/put.go:137 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in read-only mode"}
warn engine/engine.go:142 could not select objects from shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in degraded mode"}
warn engine/engine.go:142 can't perform `TreeGetOpLog` {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "pilorama is in a degraded mode"}
it seems pretty strange to me that attempts to rewrite files are considered as blunders and lead to the shard's degradation. Lets analyze can shard feel OK to object exists write exceptions and not degrade.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the error-threshold handling in engine/engine.go, then trace the shard mode transitions in shard/mode.go and the object-write path in engine/put.go. Reproduce the existing-file failure and determine whether it should be treated as harmless; done means repeated file-exists errors no longer incorrectly move the shard to degraded read-only mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100