linkedin / linkedin/ambry

Determine feasibility of avoiding copies while creating PutChunks

Open
#397 0 comments 0 reactions 1 assignee Claimed by @pnarayanan View on GitHub
enhancement
Dominant language
Java
Stars
1.8k
Forks
298
Avg merge
21h 43m
Merged PRs (30d)
9

Description

Creating this issue to explore the possibility of not creating a copy of the data while creating chunks that need to be put (at the `NonBlockingRouter`).

Currently, creating put chunks involves copying data from the buffer provided from the `ReadableStreamChannel` (the chunks themselves are implementations of `AsyncWritableChannel`). Since `ReadableStreamChannel` implementations are not allowed to overwrite/reuse buffers until they receive callbacks from the `AsyncWritableChannel`, it might be possible to avoid the copy.

The main problem will be the fact that the chunk size enforced at the `NonBlockingRouter` will not be equal to the sizes of the individual buffers received. This means that
P.1) Composite byte buffers will have to be created.
P.2) The modules downstream will have to understand these composite byte buffers (i.e. network modules).
P.3) There will be many boundary conditions (for e.g. a source buffer's data might be across chunk boundaries).

It will be useful to evaluate if this is possible (without introducing too much complexity) since avoiding the copy might translate into better performance.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.