influxdata / influxdata/kapacitor
Join with fill('null') should not deadlock
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I realize there is the following in https://docs.influxdata.com/kapacitor/v1.2/nodes/stats_node/:
> WARNING: It is not recommended to join the stats stream with the original data stream. Since they operate on different clocks you could potentially create a deadlock. This is a limitation of the current implementation and may be removed in the future.
However it wasn't clear to me whether that also applied for joins with the `.fill('null')` chain method or not. I also wasn't able to find the github issue that tracked the fix either way.
Below is a TICKscript and write script that can reproduce the issue somewhat reliably on Kapacitor v1.2.1 (git: 4628bda2e3fbcc43df694707ec3f640f213cb0dc):
TICKscript:
```
var from = stream
|from()
.measurement('test')
var stats = from
|stats(1s)
stats
|join(from)
.as('stats', 'from')
.tolerance(1s)
.fill('null')
|log()
```
Write script:
```
#!/usr/bin/env bash
nanos=$(date +%s)000000000
curl -i -XPOST 'http://localhost:9092/write?db=test&rp=default' --data-binary "test f1=0,f2=\"v2\""
sleep 1s
for i in `seq 1 10`;do
timestamp=$((nanos+i))
curl -i -XPOST 'http://localhost:9092/write?db=test&rp=default' --data-binary "test f1=$i,f2=\"v2\" $timestamp"
done
```
The output before it deadlocks:
```
[httpd] ::1 - - [20/Apr/2017:21:46:49 -0700] "POST /write?db=test&rp=default HTTP/1.1" 204 0 "-" "curl/7.51.0" 88421c89-264d-11e7-83e3-000000000000 91
[bug:log6] 2017/04/20 21:46:49 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:47Z"}
[bug:log6] 2017/04/20 21:46:50 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:48Z"}
[bug:log6] 2017/04/20 21:46:51 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:49Z"}
[bug:log6] 2017/04/20 21:46:52 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.f1":0,"from.f2":"v2","stats.emitted":0},"Time":"2017-04-21T04:46:50Z"}
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the deadlock with the provided TICKscript and bash write script on the Kapacitor v1.2.1 behavior described. Trace the join, stats, and fill('null') processing involved in the hang; done means the same input completes without deadlocking and produces the expected joined output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, go
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100