RestClientTransport sends bulk bodies as one HTTP chunk / TLS record / syscall per NDJSON buffer, burning ~45x the reactor CPU of the HLRC

## Java API client version

8.19.19 (`co.elastic.clients:elasticsearch-java`), with `org.elasticsearch.client:elasticsearch-rest-client` 8.19.19 (httpasyncclient 4.1.5, httpcore-nio 4.4.16). The code paths below are unchanged on `main` as of 2026-09-08, and `Rest5ClientTransport` has the equivalent multi-buffer entity, so this is not specific to 8.19.

## Java version

Temurin 21.0.11 (aarch64, Linux). Also reproduced on 17.

## Elasticsearch Version

8.19.9 (Elastic Cloud), TLS 1.3, AES-GCM.

## Problem description

When a `BulkRequest` is sent through `RestClientTransport`, the request body reaches Apache HttpAsyncClient as a list of small `ByteBuffer`s (one per NDJSON line plus a shared 1-byte `"\n"` separator, i.e. four buffers per index operation), and the entity that wraps them writes **one buffer per `produceContent()` call**. httpcore-nio calls `produceContent()` once per writable socket event and the chunk encoder frames each call as its own HTTP chunk. Every small buffer therefore becomes its own HTTP chunk, its own TLS record and its own `write()` syscall plus `epoll` wake-up.

For a typical 166 KB bulk of ~1,870 small documents that is roughly **7,500 TLS records and syscalls per request**, versus about 20 for the same body sent by the 7.17 High Level REST Client, which used a single `NByteArrayEntity` with `Content-Length` set.

### Where it happens

1. `ElasticsearchTransportBase.collectNdJsonLines` (8.19.19 sources, lines 297–309) adds each serialized item as its own `ByteBuffer` and then adds the shared static `NdJsonSeparator` buffer after every item.
2. `RestClientHttpClient.createRestRequest` (line 173) wraps that `Iterable` in `new MultiBufferEntity(body, ct)`.
3. `MultiBufferEntity` (line 49) calls `setChunked(true)` and reports `getContentLength() == -1`; its `produceContent()` (lines 104–120) does a single `encoder.write(currentBuffer)` and returns, only advancing to the next buffer when the current one is drained:

```java
public void produceContent(ContentEncoder encoder, IOControl ioControl) throws IOException {
if (currentBuffer == null) { encoder.complete(); return; }
encoder.write(currentBuffer); // one buffer per call
if (!currentBuffer.hasRemaining()) {
if (iterator.hasNext()) currentBuffer = iterator.next().duplicate();
else { currentBuffer = null; encoder.complete(); }
}
}
```

4. httpcore-nio's `HttpAsyncRequestExecutor.outputReady` calls `produceContent()` once per writable event, and `ChunkEncoder.write` frames each call as one chunk. With `SSLIOSession` that chunk becomes one TLS record.

The per-item `ByteBuffer`s are small (tens to a few hundred bytes for typical documents), so the encoder's session buffer is never filled; each event ships a handful of bytes.

### Measured impact

An application that migrated its bulk indexing path from HLRC 7.17.24 to elasticsearch-java 8.19.19 A/B tested both clients on the same 3-core host against the same Elastic Cloud deployment, same input stream, same starting backlog, freshly created index each time, driving each client to a complete drain of the same fixed backlog:

| | HLRC 7.17.24 | elasticsearch-java 8.19.19 |
|---|---|---|
| Sustained indexing rate | 39–44K docs/s | 20–22K docs/s, CPU-bound |
| Container CPU | 0.8–1.2 cores | 2.6–2.9 cores (87–98% of a 3-core request) |
| CPU per document | ~19–30 µs | ~121–147 µs |
| Application thread CPU per document | 15.4 µs | 16.6 µs (equal) |
| Apache HttpAsyncClient `I/O dispatcher` threads (4) | ~2.2 µs/doc, ~0.09 cores | **~115 µs/doc, ~2.4 cores** |
| Time to fully drain a fixed ~124.5M-document backlog | ~50 min | ~101–102 min |

The application-side cost was identical; the entire excess was on the four I/O reactor threads. A 60 s async-profiler capture placed 85% of all samples on those threads, of which about 87% (75% of the entire process) were under the socket write path (`AbstractIODispatch.outputReady` → `SSLIOSession.outboundTransport` → `SSLEngineImpl.wrap` / `GaloisCounterMode` → `SocketChannelImpl.write` → `EPoll.wait`), with `MultiBufferEntity.produceContent` → `ChunkEncoder.write` at the top of the tree. `BulkIngester`, JSON serialization and response decoding were each under 1%.

### Confirmation

Wrapping the `TransportHttpClient` with a decorator that merges the body into a single `ByteBuffer` before delegating to `RestClientHttpClient` (bytes on the wire identical, only the chunk boundaries change) brought the reactor threads from 2.5 live cores down to 0.15–0.20 (about 4 µs/doc — still somewhat higher than HLRC's ~2.2 µs/doc, but now a small fraction of total cost), container CPU from ~2.9 pinned down to 0.8–1.0 cores, and the rate back up to 39–42K docs/s, matching HLRC's own 39–44K. Total client CPU per document dropped from ~132 µs to ~19 µs — at or below HLRC's own ~19–30 µs, since HLRC pays an additional cost decoding bulk responses on its own dedicated thread pool that this client's architecture doesn't require. Over a complete drain of the same fixed backlog, total time closed from roughly 2× HLRC's down to matching it almost exactly (~51–52 min vs HLRC's ~50 min).

```java
// Decorator used to confirm; deployed application-side as a workaround.
static Request coalesce(Request r) {
Iterable body = r.body();
if (body == null) return r;
List buffers = new ArrayList<>(); int size = 0;
for (ByteBuffer b : body) { buffers.add(b); size += b.remaining(); }
if (buffers.size() <= 1) return r;
ByteBuffer merged = ByteBuffer.allocate(size);
for (ByteBuffer b : buffers) merged.put(b.duplicate()); // NdJsonSeparator is shared; never advance it
merged.flip();
return new Request(r.method(), r.path(), r.queryParams(), r.headers(), Collections.singletonList(merged));
}
```

`RestClient`'s request compression (`setCompressionEnabled(true)`) masks the issue as a side effect because `ContentCompressingEntity` gzips the whole body into one array, but it moves gzip CPU onto the same reactor threads, so it is a mitigation rather than a fix.

## Suggested fix

Any of these in `MultiBufferEntity.produceContent()` would restore the HLRC's framing without changing the public API:

1. **Loop until the encoder stops accepting bytes**: keep calling `encoder.write(currentBuffer)` and advancing buffers while `encoder.write` returns > 0 and buffers remain, so each writable event fills the session buffer rather than shipping one small buffer.
2. **Report `Content-Length`** when the total size is known (it is: the buffers are already materialized), so `LengthDelimitedEncoder` is used instead of `ChunkEncoder` and the entity can be written in large slices.
3. **Coalesce in `collectNdJsonLines`** (fewer, larger buffers), though (1) or (2) fixes every multi-buffer body, not just NDJSON.

Option (1) is the smallest change and is what `EntityAsyncContentProducer` effectively does for a byte-array entity (it reads 4 KB slices in a loop per event).

## Steps to reproduce

1. Build a `RestClientTransport` over a `RestClient` pointing at any TLS-terminated Elasticsearch (Elastic Cloud reproduces cleanly).
2. Drive `BulkIngester` (or call `ElasticsearchAsyncClient.bulk` directly) with bulks of ~1,500–2,000 small (~100 byte) documents, enough concurrency to saturate one CPU core.
3. Take a `jstack` and compare the cumulative `cpu=` field of the `I/O dispatcher N` threads against the application threads, or run async-profiler in `itimer` mode; the reactor threads dominate and the tree is `MultiBufferEntity.produceContent → ChunkEncoder.write → SSLIOSession → write()`.
4. Optionally `strace -c -f -e write` the JVM: the `write` count per bulk is in the thousands.
5. Apply the coalescing decorator above at the `TransportHttpClient` seam and repeat; the reactor CPU drops by ~20×.

I can share the flamegraph HTML and thread dumps on request.