Improvements for handling many small entries
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
The BK data path is very efficient when processing large entries and it's generally able to saturate the disk and network IO in these cases.
By contrast, when handling a large number of very small entries there are several inefficiencies that cause the CPU to become the bottleneck, because of the per-entry overhead.
There are several low-hanging fruits to tackle to improve performance:
#### Reduce contention between message passing
Reduce contention in journal & force-write queues:
- [x] #3544
- [x] #3545
Improve the OrderedExecutor performance:
- [x] #3546
#### Reduce the number of buffers allocated per entry written/read.
For each entry being written in a ledger we are using 4 `ByteBuf` instances:
1. The entry payload (this gets passed in to BK client)
2. The checksum
3. The serialized `AddRequest
4. The 4 byte size header
These buffers are passed to Netty which will do a scatter `writev`, though it will pass all the buffers.
Allocating and managing all these buffer is expensive. There is overhead in:
* Refcounting
* Recycler to get the `ByteBuf` instances and put them back in the pool
* ByteBuf pool arena to handle allocations/deallocation
* Inter-thread synchronization: these buffer are normally allocated in one thread and deallocated from a different thread
To make matters worse, while the checksum is computed only once, the `AddRequest` is serialized each time we write it on a connection.
eg: if we have write-quorum=3, it would mean we are using (2 * 3) + 1 = 7 `ByteBuf` per each entry.
Finally, while for big entries is very important to avoid copying the payload, for small entries the overhead of maintaining the `ByteBufList` is greater than just copying the payload into a single buffer.
For that we should do:
1. If the entries are big -> keep using `ByteBufList`, with 1 buffer for all the header and the 2nd buffer referencing the payload, with no copy.
2. If entry is small -> allocate a buffer to contain all the headers and the payload and copy into it.
Pending changes:
- [x] #3560 Add the 4 bytes frame size header when serializing the request, instead of relying on a separate Netty filter
- [ ] Consolidate buffer for small entries on read-response
- [ ] Serialize only once and consolidate small entries for add requests
Contributor guide
Assessment
This issue has not been assessed yet.