influxdata / influxdata/kapacitor

Join with fill('null') should not deadlock

Open
#1,334 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I realize there is the following in https://docs.influxdata.com/kapacitor/v1.2/nodes/stats_node/:

> WARNING: It is not recommended to join the stats stream with the original data stream. Since they operate on different clocks you could potentially create a deadlock. This is a limitation of the current implementation and may be removed in the future.

However it wasn't clear to me whether that also applied for joins with the `.fill('null')` chain method or not. I also wasn't able to find the github issue that tracked the fix either way.

Below is a TICKscript and write script that can reproduce the issue somewhat reliably on Kapacitor v1.2.1 (git: 4628bda2e3fbcc43df694707ec3f640f213cb0dc):

TICKscript:
```
var from = stream
|from()
.measurement('test')

var stats = from
|stats(1s)

stats
|join(from)
.as('stats', 'from')
.tolerance(1s)
.fill('null')
|log()
```
Write script:
```
#!/usr/bin/env bash
nanos=$(date +%s)000000000
curl -i -XPOST 'http://localhost:9092/write?db=test&rp=default' --data-binary "test f1=0,f2=\"v2\""
sleep 1s
for i in `seq 1 10`;do
timestamp=$((nanos+i))
curl -i -XPOST 'http://localhost:9092/write?db=test&rp=default' --data-binary "test f1=$i,f2=\"v2\" $timestamp"
done
```
The output before it deadlocks:
```
[httpd] ::1 - - [20/Apr/2017:21:46:49 -0700] "POST /write?db=test&rp=default HTTP/1.1" 204 0 "-" "curl/7.51.0" 88421c89-264d-11e7-83e3-000000000000 91
[bug:log6] 2017/04/20 21:46:49 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:47Z"}

[bug:log6] 2017/04/20 21:46:50 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:48Z"}

[bug:log6] 2017/04/20 21:46:51 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.emitted":null,"stats.emitted":0},"Time":"2017-04-21T04:46:49Z"}

[bug:log6] 2017/04/20 21:46:52 I! {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{},"Fields":{"from.f1":0,"from.f2":"v2","stats.emitted":0},"Time":"2017-04-21T04:46:50Z"}
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the deadlock with the provided TICKscript and bash write script on the Kapacitor v1.2.1 behavior described. Trace the join, stats, and fill('null') processing involved in the hang; done means the same input completes without deadlocking and produces the expected joined output.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, go
Domain
backend, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.