nspcc-dev / nspcc-dev/neofs-node
object/put: Use one data buffer per-request
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
Is your feature request related to a problem? Please describe.
ObjectService.Put devours a lot of memory for data buffers, sometimes excessively. Current scheme:
client ---> server OS |---> gRPC ---> [slicer] ---> ObjectService ---> storage
- node process cannot affect anything up to OS overall
- no slicer when object is "ready"
each stage component of the whole processing buffers the data. This leads to redundancy leading to memory overuse
Describe the solution you'd like
allocate single buffer for the whole operation
OS |---> [xxx__buffer__xxx] ---> storage
where, in general, storage are data drives that could be remote (depending on the client-defined storage policy)
The buffer size depends on the following factors:
- fixed amount of data specified by the client (undefined in general)
- optimal transmission unit of local and remote drives
- optimal transmission unit b/w all remote node connections
MaxObjectSize- current node state
node selects optimal buffer size and serves request within the allocated buffer. Then algorithm is simple:
- read buffer from
we can start from the simplest min(MaxObjectSize, PayloadSize | Inf) model and get rid of redundant data copies. This will already reduce resource consumption horizontally. Then we'll develop more adaptive models
Describe alternatives you've considered
not yet
Additional context
- #2686
- #2719
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/services/object/put/distributed.go at the linked ObjectService.Put path, then read the related context in issues #2686 and #2719. Establish how the initial min(MaxObjectSize, PayloadSize | Inf) buffer model fits the request flow and how reduced copying would be verified; the issue does not name tests or a complete implementation boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100