influxdata / influxdata/influxdb

Join operation within two streams limited to 2000 rows

Open
#24,065 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hi Everyone,

I am struggling over what really seems to be a bug.
In my situation I have two streams:

1. stream A having the energy used by a machine, sampled 2880 times per day;
2. stream B having the energy cost sampled once per day;

In order to compute the cost at any of the 2880 timings per day (I need a time series for it for other purposes) what I do is:

1. I create a '_day' column for both the stream A and B storing the day timestamp, I then group both by this new '_day' column;
2. I then take a LEFT join on stream A and B over the '_day' column in order to obtain a unique stream and be able to compute the energy cost for any of the timings;

This approach works indeed, the very subtle problem is that if you have more than N>2000 rows for stream A, for some reason the joined table will return N-1000 rows.

So for a single day the joined table will return 1880 rows instead of 2880! If i limit the stream A to 2000 rows I will get 2000 rows back, but if I use 2001 rows I will get back 1001 rows.

In this image _value -> is the count of rows.

![image](https://user-images.githubusercontent.com/32195584/214906802-f16aa294-0c23-416d-ac5b-ab1a00c06e87.png)

Thanks to anyone helping me.

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Create a stream A with at least 2001 rows per day;
2. Add the _day column to stream A containing the day timestamp -> date.truncate(t: r._time, unit 1d)
3. Create a stream B with 1 row per day;
4. Add the _day column to stream B containing the day timestamp -> date.truncate(t: r._time, unit 1d)
5. Take a letf/full join on the _day column

__Expected behavior:__
Get back a table having the same row numbers than the stream A.

__Actual behavior:__

Get back a table having the same row numbers than the stream A only if row_numbers <= 2000.

__Environment info:__

* System info: InfluxDB OSS running on docker container
* InfluxDB version: 2.6.1

Contributor guide

Open the contributing guide

Research direction

Reproduce the join using InfluxDB OSS 2.6.1 in Docker with stream A containing more than 2,000 rows per day and stream B containing one row per day. Verify the _day grouping and left/full join steps, then confirm that the result preserves every stream A row rather than dropping 1,000 rows.

Written by the indexing model from the issue text.

Assessment

Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.