influxdata / influxdata/influxdb-java

Proposal: Separate high performance point writer

Aperta
#294 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
kind/enhancement
Lingua principale
Java
Stelle
1.2k
Fork
469
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

As I started discussing in #289, I'm building an InfluxDB producer that I'd like to guarantee at-least-once delivery for. The intended use is in a streaming context where I need to guarantee all records I've received up to a certain point have been durably persisted and must handle a rate of records such that writes must be batched to keep up.

I couldn't directly use the existing automatic batching capabilities of the `InfluxDBImpl` and `BatchProcessor` because I had no visibility into when a point was persisted. After reading the related issues and PRs here (#148 and #108 specifically) I decided the best approach for my use case was to implement a client that would automatically batch records but also gave me the ability to be notified when an individual point's batch write was complete - successful or not. It uses an `InfluxDB` client to send batches of points as quickly as possible. Points are grouped in to as few batch writes as possible and are flushed at a configurable interval and whenever the buffer reaches a specific capacity.

Is this something you'd like to see contributed to this repository or should it live on its own?

Here's the interface:

```
/**
* Asynchronous InfluxDB client that accumulates individual point writes into batches.
*/
public interface BatchAccumulatingInfluxDBClient {

/**
* Asynchronously queue a point to be written and invoke the provided callback after the point has been written. This may block if the underlying
* implementation cannot buffer the point immediately. The provided callback will be invoked after the attempt to write the batch finishes and before the
* future that was returned at invocation time is completed.
*
* @param database The database to write the point to.
* @param retentionPolicy The retention policy for the point.
* @param point The point to write.
* @param callback A callback to invoke on successful write or when an error was encountered while attempting to write the record.
* @return A future indicating the status of the asynchronous operation.
* @throws InterruptedException if the thread is interrupted while blocking.
* @throws TimeoutException if the write needs to block and has to wait longer then the configured timeout to enqueue the write.
*/
Future write(String database, String retentionPolicy, InfluxDB.ConsistencyLevel consistencyLevel, Point point, WriteCallback callback)
throws InterruptedException, TimeoutException;

/**
* Triggers all pending batched records to be sent and blocks until the completion of the writes.
*
* @throws InterruptedException if the thread is interrupted while blocking.
*/
void flush() throws InterruptedException;
}
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Iniziare esaminando InfluxDBImpl e BatchProcessor, quindi esaminare le issue e le pull request correlate #289, #148 e #108. Una proposta completa definirebbe e implementerebbe il client di batching separato, includendo un callback o il completamento di un future dopo ogni tentativo di batch, oltre a un’operazione di flush bloccante con intervallo e capacità configurabili.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
databases
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.