nspcc-dev / nspcc-dev/neofs-node
Accessing local storage outside the container when reading objects
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
Is your feature request related to a problem? Please describe.
currently, SN object read (GET/HEAD/RANGE) handler always accesses local storage. If the object is present, SN does not access the container members
normally, SN outside the container does not store its objects on a permanent basis. There is a transitional state when SN, after the epoch change, leaves the container and begins to migrate data to a new location. The faster this is done, the more efficiently the cluster functions
in total:
- local lookup makes sense, and sometimes even necessary
- hitting an object is either completely impossible or possible, but the probability drops rapidly over time
i think all this makes unconditional touching of local storage impractical
Describe the solution you'd like
manage list of to-migrate containers for which SN stores data locally but does not belong to. Iff requested container is non-local and to-migrate one, skip local read
the list is update on:
- SN init
- new epoch in runtime
- container migration finish
having this list, whole migration procedure may go faster. Afaik it is object-by-object now
Describe alternatives you've considered
no for now
Additional context
i've run 10m load to GET object of 4K payload size from REP 1 container through non-container SN. Here's cpu.pb.gz profile of proxy node. As we can see, local execution takes ~20% time
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the SN GET/HEAD/RANGE read handler and the lifecycle points named in the issue: SN initialization, new epochs in runtime, and migration completion. Define how the to-migrate container list is maintained and verify that reads for non-local containers skip local storage while other reads retain current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100