nspcc-dev / nspcc-dev/neofs-node

shard: Blobstor `file exists` errors move shard into degraded mode

Open
#2,438 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion I4 neofs-storage S4 U3
Dominant language
Go
Stars
38
Forks
51
Avg merge
1d 20h
Merged PRs (30d)
38

Description

looks like a bug but needs to be investigated. When local storage is loaded by objects with existing files in blobstor, shard shards switches to degraded mode fast and stops storage service

warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 1, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 2, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 3, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 4, "error": "could not put object to BLOB storage: file exists"}
...
info        shard/mode.go:28        setting shard mode        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "old_mode": "READ_WRITE", "new_mode": "DEGRADED_READ_ONLY"}
info        shard/mode.go:69        shard mode set successfully        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "mode": "DEGRADED_READ_ONLY"}
info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 30}
...
warn        engine/put.go:137        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in read-only mode"}
warn        engine/engine.go:142        could not select objects from shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in degraded mode"}
warn        engine/engine.go:142        can't perform `TreeGetOpLog`        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "pilorama is in a degraded mode"}

it seems pretty strange to me that attempts to rewrite files are considered as blunders and lead to the shard's degradation. Lets analyze can shard feel OK to object exists write exceptions and not degrade.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the error-threshold handling in engine/engine.go, then trace the shard mode transitions in shard/mode.go and the object-write path in engine/put.go. Reproduce the existing-file failure and determine whether it should be treated as harmless; done means repeated file-exists errors no longer incorrectly move the shard to degraded read-only mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.