BatchProcessor, going single to multithread ?

Using influxdb-java on a 4core/8thread machine to connect to a remote InfluxDB (located in the same DC), I experience very high CPU usage for the thread responsible for the BatchProcessor and suspect we could benefit from having BatchProcess use several threads.

It looks like all batched points go to a unique queue in the BatchProcessor instance and that a singleThreadScheduledExecutor() is responsible for dequeuing them.

My setup is not CPU bound (as shown by htop), nor memory nor network bound. Do you think allowing parallel queuing/unqueuing would increase capacity of influxdb-java to flush its data ?

I'm thinking about having a configurable number of queues (let's call it N), one singleThreadScheduledExecutor() each. Because the data is shared nothing we could make the batchProcess queue the data in a round-robin way, and then have their respective scheduleExecutor do the job.

Does this make sense ? Do you think is is worth ?

Commencez par lire l’implémentation de BatchProcessor et le chemin singleThreadScheduledExecutor décrit dans l’issue. Déterminez si des files d’attente et des exécuteurs parallèles configurables sont réalisables sans modifier le comportement du batching, puis définissez des mesures ou des tests qui montreraient si la modification augmente la capacité de flush.

Rédigé par le modèle d'indexation à partir du texte de l'issue.