influxdata / influxdata/influxdb-java

Immediate query after batch write misses points

Open
#584 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.2k
Forks
469
PR merge metrics
No merged PRs in 30d

Description

Querying for points immediately after batch write misses points. Example:

  @Test
  public void test()
  {
    // Query the count of points
    Query query = new Query("select count(disk)from disk", databaseName);

    // Points to write
    int points = BatchOptions.DEFAULTS.getActions() + 1;

    db.enableBatch(BatchOptions.DEFAULTS);

    long time = System.currentTimeMillis();
    for(int i = 0; i < points; i++)
    {
      db.write(Point.measurement("disk")
                   .time(time + i, TimeUnit.MILLISECONDS)
                   .addField("disk", 1)
                   .build());
    }

    // Flush final points to influx
    db.disableBatch();

    QueryResult result = db.query(query);
    assertEquals(points, result.getResults().get(0).getSeries().get(0).getValues().size());
  }

This unit test fails the assert statement saying "Excepted: 1001, Actual: 1". Manually inspecting the database afterwards shows the count query returning all 1001 points as expected.

Here's my influxdb log showing what's happening (sorted by time descending)

Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "POST /write?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&rp=autogen&precision=n&consistency=one HTTP/1.1" 204 0 "-" "okhttp/3.5.0" ab901640-4c18-11e9-8c02-00505684bea9 42142
Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "GET /query?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&q=select+count%28disk%29from+disk HTTP/1.1" 200 125 "-" "okhttp/3.5.0" ab8e7edf-4c18-11e9-8c01-00505684bea9 769
Mar 21 20:34:05 mschwore71 influxd[21484]: ts=2019-03-21T20:34:05.268221Z lvl=info msg="Executing query" log_id=0EKIBsll000 service=query query="SELECT count(disk) FROM influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a.autogen.disk"
Mar 21 20:34:05 mschwore71 influxd[21484]: [httpd] 127.0.0.1 - null [21/Mar/2019:20:34:05 +0000] "POST /write?db=influx_tests_c7ccabbe_7bb7_404a_85b7_4d3a9531fb6a&rp=autogen&precision=n&consistency=one HTTP/1.1" 204 0 "-" "okhttp/3.5.0" ab8b607d-4c18-11e9-8c00-00505684bea9 13274

Stepping through the debugger, I see something like this happening (on my machine at least)

  1. [Main Thread] Write buffer is filled to 1k points
  2. [Main Thread] Schedule worker thread to write 1k points
  3. [Main Thread] Another 1 point is added to the write buffer
  4. [Main Thread] Disable batch
  5. [Main Thread] Write remaining 1 point in buffer
  6. [Main Thread] Query for points
  7. [Worker Thread] Write 1k points

Also to note, this causes influxdb-java to write points out of order to influxdb.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided unit-test scenario with BatchOptions.DEFAULTS, then inspect the batch worker and disableBatch behavior involved in flushing points. Verify that disabling batching waits for the scheduled 1,000-point write before the immediate query runs, and that the query returns all 1,001 points without out-of-order writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.