nspcc-dev / nspcc-dev/neofs-node
Drop pools from local shards
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
Is your feature request related to a problem? Please describe.
I'm always frustrated when we have limits in the system that are hard to explain/configure/have any meaningful defaults for. Shard pools are exactly that, they limit concurrency artificially, this limit has no relation to real system capabilities (for example, SSDs used for storage) and product technologies (FSTree combined writer can handle a lot of objects, but we're only giving it like eight).
Describe the solution you'd like
Have a deadline for write operation. Push data into shard as it goes. Collect real operation times observed in a sliding window manner (clustered by size, potentially). Predict operation time based on statistics collected. Start dropping some requests with ErrBusy once predicted time is >80% (60-70-90, depends) of the target one, drop more of them as time grows.
Describe alternatives you've considered
None.
Additional context
Related to #2479 (can also take context deadline into account)?
See https://en.wikipedia.org/wiki/CoDel for inspiration.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading related issue #2479 and the current shard-pool implementation, then review CoDel for the proposed deadline and queue-management ideas. Done means replacing the fixed local shard pools with deadline-aware behavior that uses observed operation times and returns ErrBusy as the target time is approached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100