apache / apache/bookkeeper

Improvements for handling many small entries

Open
#3,559 2 comments 7 reactions 0 assignees View on GitHub
type/feature
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

The BK data path is very efficient when processing large entries and it's generally able to saturate the disk and network IO in these cases.

By contrast, when handling a large number of very small entries there are several inefficiencies that cause the CPU to become the bottleneck, because of the per-entry overhead.

There are several low-hanging fruits to tackle to improve performance:

#### Reduce contention between message passing

Reduce contention in journal & force-write queues:
- [x] #3544
- [x] #3545

Improve the OrderedExecutor performance:
- [x] #3546

#### Reduce the number of buffers allocated per entry written/read.

For each entry being written in a ledger we are using 4 `ByteBuf` instances:
1. The entry payload (this gets passed in to BK client)
2. The checksum
3. The serialized `AddRequest
4. The 4 byte size header

These buffers are passed to Netty which will do a scatter `writev`, though it will pass all the buffers.
Allocating and managing all these buffer is expensive. There is overhead in:
* Refcounting
* Recycler to get the `ByteBuf` instances and put them back in the pool
* ByteBuf pool arena to handle allocations/deallocation
* Inter-thread synchronization: these buffer are normally allocated in one thread and deallocated from a different thread

To make matters worse, while the checksum is computed only once, the `AddRequest` is serialized each time we write it on a connection.
eg: if we have write-quorum=3, it would mean we are using (2 * 3) + 1 = 7 `ByteBuf` per each entry.

Finally, while for big entries is very important to avoid copying the payload, for small entries the overhead of maintaining the `ByteBufList` is greater than just copying the payload into a single buffer.
For that we should do:
1. If the entries are big -> keep using `ByteBufList`, with 1 buffer for all the header and the 2nd buffer referencing the payload, with no copy.
2. If entry is small -> allocate a buffer to contain all the headers and the payload and copy into it.

Pending changes:
- [x] #3560 Add the 4 bytes frame size header when serializing the request, instead of relying on a separate Netty filter
- [ ] Consolidate buffer for small entries on read-response
- [ ] Serialize only once and consolidate small entries for add requests

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.