nspcc-dev / nspcc-dev/neofs-node

Drop pools from local shards

Open
#3,985 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement I3 neofs-storage performance S3 U2
Dominant language
Go
Stars
38
Forks
51
Avg merge
1d 20h
Merged PRs (30d)
38

Description

Is your feature request related to a problem? Please describe.

I'm always frustrated when we have limits in the system that are hard to explain/configure/have any meaningful defaults for. Shard pools are exactly that, they limit concurrency artificially, this limit has no relation to real system capabilities (for example, SSDs used for storage) and product technologies (FSTree combined writer can handle a lot of objects, but we're only giving it like eight).

Describe the solution you'd like

Have a deadline for write operation. Push data into shard as it goes. Collect real operation times observed in a sliding window manner (clustered by size, potentially). Predict operation time based on statistics collected. Start dropping some requests with ErrBusy once predicted time is >80% (60-70-90, depends) of the target one, drop more of them as time grows.

Describe alternatives you've considered

None.

Additional context

Related to #2479 (can also take context deadline into account)?

See https://en.wikipedia.org/wiki/CoDel for inspiration.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading related issue #2479 and the current shard-pool implementation, then review CoDel for the proposed deadline and queue-management ideas. Done means replacing the fixed local shard pools with deadline-aware behavior that uses observed operation times and returns ErrBusy as the target time is approached.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.