[FEATURE] Adaptive maxOperations for BulkInjester

Is your feature request related to a problem?

BulkIngester requires users to manually tune maxOperations or maxByteSize to achieve optimal indexing throughput. The optimal value depends on cluster capacity, document size, shard count, and current load — all of which vary at runtime and differ across deployments. Setting it too low leaves throughput on the table; setting it too high can cause memory pressure or trigger 429s from an overloaded cluster. Users currently have to run offline benchmarks, pick a static number, and hope it holds under production conditions.

What solution would you like?

An optional AdaptiveOperationsController that automatically tunes maxOperations at runtime by observing BulkResponse.took() after every bulk response. Users opt in with one line:

BulkIngester ingester = BulkIngester.of(b -> b
.client(client)
.adaptiveOperations(new AdaptiveOperationsController(200)) // initialOps: 200, optional
);

The controller finds the throughput-optimal maxOperations without any manual tuning. initialOps can be optional set as an initial value for maxOperations; the controller takes over from the first response onward.

Algorithm: Hill Climbing + AIMD Decrease

Two mechanisms handle two separate signals:

• Hill climbing (increase path) — after each bulk response, computes throughput = batchSize / took * 1000 (ops/s), smoothed with EWMA. Increases maxOperations by a step while throughput keeps improving. Stops at the plateau where further increases yield less than a configurable minimum gain (default 0.5%). Step size halves on each regression, enabling coarse-grained ramp-up and fine-grained search near the optimum.

• AIMD decrease (overload path) — when 429s or transport failures are detected, immediately multiplies maxOperations by a decrease factor (default 0.75) and resets the hill climb from the new lower baseline.

Controller trace:
ops=200 → improving → ops=400
ops=400 → improving → ops=600
... (fast ramp, step=initialOps)
ops=5000 → improving → ops=5200
ops=5200 → gain=0.04% < 0.5% → PLATEAU → hold at ops=5000

What alternatives have you considered?

Users do multiple rounds tests to find the optimal value for the maxOperations parameter or maxSize parameter against a cluster having same configuration with the production cluster, and then change the client code to apply that value, this should be done again when switching to another OpenSearch cluster.

Start with the BulkIngester entry point and BulkResponse.took(); map how maxOperations and failures are currently handled. Done means an optional AdaptiveOperationsController can accept an initial value, adapt after responses, and respond to 429s or transport failures without changing default behavior.