influxdata / influxdata/influxdb-java
Immediate query after batch write misses points
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 469
- PR merge metrics
- No merged PRs in 30d
Description
Querying for points immediately after batch write misses points. Example:
@Test
public void test()
{
// Query the count of points
Query query = new Query("select count(disk)from disk", databaseName);
// Points to write
int points = BatchOptions.DEFAULTS.getActions() + 1;
db.enableBatch(BatchOptions.DEFAULTS);
long time = System.currentTimeMillis();
for(int i = 0; i < points; i++)
{
db.write(Point.measurement("disk")
.time(time + i, TimeUnit.MILLISECONDS)
.addField("disk", 1)
.build());
}
// Flush final points to influx
db.disableBatch();
QueryResult result = db.query(query);
assertEquals(points, result.getResults().get(0).getSeries().get(0).getValues().size());
}
This unit test fails the assert statement saying "Excepted: 1001, Actual: 1". Manually inspecting the database afterwards shows the count query returning all 1001 points as expected.
Here's my influxdb log showing what's happening (sorted by time descending)
Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "POST /write?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&rp=autogen&precision=n&consistency=one HTTP/1.1" 204 0 "-" "okhttp/3.5.0" ab901640-4c18-11e9-8c02-00505684bea9 42142
Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "GET /query?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&q=select+count%28disk%29from+disk HTTP/1.1" 200 125 "-" "okhttp/3.5.0" ab8e7edf-4c18-11e9-8c01-00505684bea9 769
Mar 21 20:34:05 mschwore71 influxd[21484]: ts=2019-03-21T20:34:05.268221Z lvl=info msg="Executing query" log_id=0EKIBsll000 service=query query="SELECT count(disk) FROM influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a.autogen.disk"
Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "POST /write?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&rp=autogen&precision=n&consistency=one HTTP/1.1" 204 0 "-" "okhttp/3.5.0" ab8b607d-4c18-11e9-8c00-00505684bea9 13274
Stepping through the debugger, I see something like this happening (on my machine at least)
- [Main Thread] Write buffer is filled to 1k points
- [Main Thread] Schedule worker thread to write 1k points
- [Main Thread] Another 1 point is added to the write buffer
- [Main Thread] Disable batch
- [Main Thread] Write remaining 1 point in buffer
- [Main Thread] Query for points
- [Worker Thread] Write 1k points
Also to note, this causes influxdb-java to write points out of order to influxdb.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided unit-test scenario with BatchOptions.DEFAULTS, then inspect the batch worker and disableBatch behavior involved in flushing points. Verify that disabling batching waits for the scheduled 1,000-point write before the immediate query runs, and that the query returns all 1,001 points without out-of-order writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100