influxdata / influxdata/influxdb-java
Proposal: Separate high performance point writer
- Lingua principale
- Java
- Stelle
- 1.2k
- Fork
- 469
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
As I started discussing in #289, I'm building an InfluxDB producer that I'd like to guarantee at-least-once delivery for. The intended use is in a streaming context where I need to guarantee all records I've received up to a certain point have been durably persisted and must handle a rate of records such that writes must be batched to keep up.
I couldn't directly use the existing automatic batching capabilities of the `InfluxDBImpl` and `BatchProcessor` because I had no visibility into when a point was persisted. After reading the related issues and PRs here (#148 and #108 specifically) I decided the best approach for my use case was to implement a client that would automatically batch records but also gave me the ability to be notified when an individual point's batch write was complete - successful or not. It uses an `InfluxDB` client to send batches of points as quickly as possible. Points are grouped in to as few batch writes as possible and are flushed at a configurable interval and whenever the buffer reaches a specific capacity.
Is this something you'd like to see contributed to this repository or should it live on its own?
Here's the interface:
```
/**
* Asynchronous InfluxDB client that accumulates individual point writes into batches.
*/
public interface BatchAccumulatingInfluxDBClient {
/**
* Asynchronously queue a point to be written and invoke the provided callback after the point has been written. This may block if the underlying
* implementation cannot buffer the point immediately. The provided callback will be invoked after the attempt to write the batch finishes and before the
* future that was returned at invocation time is completed.
*
* @param database The database to write the point to.
* @param retentionPolicy The retention policy for the point.
* @param point The point to write.
* @param callback A callback to invoke on successful write or when an error was encountered while attempting to write the record.
* @return A future indicating the status of the asynchronous operation.
* @throws InterruptedException if the thread is interrupted while blocking.
* @throws TimeoutException if the write needs to block and has to wait longer then the configured timeout to enqueue the write.
*/
Future write(String database, String retentionPolicy, InfluxDB.ConsistencyLevel consistencyLevel, Point point, WriteCallback callback)
throws InterruptedException, TimeoutException;
/**
* Triggers all pending batched records to be sent and blocks until the completion of the writes.
*
* @throws InterruptedException if the thread is interrupted while blocking.
*/
void flush() throws InterruptedException;
}
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Iniziare esaminando InfluxDBImpl e BatchProcessor, quindi esaminare le issue e le pull request correlate #289, #148 e #108. Una proposta completa definirebbe e implementerebbe il client di batching separato, includendo un callback o il completamento di un future dopo ogni tentativo di batch, oltre a un’operazione di flush bloccante con intervallo e capacità configurabili.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- databases
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100