influxdata / influxdata/influxdb-java
Proposal: Separate high performance point writer
- Ngôn ngữ chính
- Java
- Star
- 1.2k
- Fork
- 469
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
As I started discussing in #289, I'm building an InfluxDB producer that I'd like to guarantee at-least-once delivery for. The intended use is in a streaming context where I need to guarantee all records I've received up to a certain point have been durably persisted and must handle a rate of records such that writes must be batched to keep up.
I couldn't directly use the existing automatic batching capabilities of the `InfluxDBImpl` and `BatchProcessor` because I had no visibility into when a point was persisted. After reading the related issues and PRs here (#148 and #108 specifically) I decided the best approach for my use case was to implement a client that would automatically batch records but also gave me the ability to be notified when an individual point's batch write was complete - successful or not. It uses an `InfluxDB` client to send batches of points as quickly as possible. Points are grouped in to as few batch writes as possible and are flushed at a configurable interval and whenever the buffer reaches a specific capacity.
Is this something you'd like to see contributed to this repository or should it live on its own?
Here's the interface:
```
/**
* Asynchronous InfluxDB client that accumulates individual point writes into batches.
*/
public interface BatchAccumulatingInfluxDBClient {
/**
* Asynchronously queue a point to be written and invoke the provided callback after the point has been written. This may block if the underlying
* implementation cannot buffer the point immediately. The provided callback will be invoked after the attempt to write the batch finishes and before the
* future that was returned at invocation time is completed.
*
* @param database The database to write the point to.
* @param retentionPolicy The retention policy for the point.
* @param point The point to write.
* @param callback A callback to invoke on successful write or when an error was encountered while attempting to write the record.
* @return A future indicating the status of the asynchronous operation.
* @throws InterruptedException if the thread is interrupted while blocking.
* @throws TimeoutException if the write needs to block and has to wait longer then the configured timeout to enqueue the write.
*/
Future write(String database, String retentionPolicy, InfluxDB.ConsistencyLevel consistencyLevel, Point point, WriteCallback callback)
throws InterruptedException, TimeoutException;
/**
* Triggers all pending batched records to be sent and blocks until the completion of the writes.
*
* @throws InterruptedException if the thread is interrupted while blocking.
*/
void flush() throws InterruptedException;
}
```
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Bắt đầu bằng cách xem xét InfluxDBImpl và BatchProcessor, sau đó kiểm tra các issue và pull request liên quan #289, #148 và #108. Một đề xuất hoàn chỉnh sẽ định nghĩa và triển khai batching client riêng biệt, bao gồm callback hoặc việc hoàn tất future sau mỗi lần thử batch, cùng với một thao tác flush blocking có khoảng thời gian và dung lượng có thể cấu hình.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- java
- Lĩnh vực
- databases
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 25/100