nspcc-dev / nspcc-dev/neofs-node
Meta: optimize indexing objects
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
It is known that it is possible to PUT objects more often than meta indexing can do: 1K queue is filled quite fast (minutes on real machines with real networks) and even 1M queue does not stand forever.
Expected Behavior
Handing objects from a block should take time that can compete block time itself. In our meta tests it is 50/100ms.
Current Behavior
Handling notifications from block takes longer than the block lasts, block queue fills quite fast.
Possible Solution
- Cache HEADers in memory as objects are stored in the storage engine
- Optimize indexing storage
- ...
Steps to Reproduce (for bugs)
Longivity PUT tests with small object sizes.
Regression
No
Your Environment
0.47.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading pkg/services/meta/meta.go around line 137 and running the longevity PUT tests with small object sizes. Measure notification handling against block duration and inspect the queue growth described in the issue. Done means block notifications are processed quickly enough to prevent the queue from filling under the reported workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100