influxdata / influxdata/influxdb

Restarting the database can result in concurrent task executions leading to failures

Open
#21,689 0 comments 0 reactions 0 assignees View on GitHub
area/2.x area/tasks kind/bug
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
1. Implement a data intensive task that takes > 5 minute to run, and define task.concurrency as 1. (this is to simulate a task that processes data from the time it last ran until present, rather than processing data only within it's window)
2. Set the interval for the task to 1 minute
3. Turn off influxdb server for an amount of time (say 30 minutes)
4. Turn it back on again

__Expected behavior:__
Catch-up tasks are scheduled for the periods when the database was offline. These execute one at a time, waiting until each task is finished before starting the next one

__Actual behavior:__
Catch-up tasks are scheduled for the periods when the database was offline. These execute at their scheduled time regardless of concurrency settings and eventually result in task failures and queue length exceeded errors.

__Environment info:__

* System info: Run `uname -srm` and copy the output here
Linux 5.4.72-microsoft-standard-WSL2 x86_64
* InfluxDB version: Run `influxd version` and copy the output here
InfluxDB 2.0.4 (git: 4e7a59bb9a) build_date: 2021-02-08T17:47:02Z
* Other relevant environment details: Container runtime, disk info, etc
Running in docker engine v20.10.6

__Config:__
```
docker run -d -p 8086:8086 --name influxdb2 \
-v $PWD:/root/.influxdbv2 \
quay.io/influxdb/influxdb:v2.0.4 \
influxd --feature-flags simpleTaskOptionsExtraction=true
```

Thanks in advance,

Pete

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by reproducing the database restart scenario with the provided Docker configuration, then trace catch-up task scheduling and the concurrency limit. Done means offline-period tasks run one at a time when concurrency is 1, without task failures or queue length exceeded errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.