influxdata / influxdata/community-templates

Downsampling Task that works on all data types without hardcoded filters

Open
#296 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Template Request
Dominant language
Python
Stars
376
Forks
163
PR merge metrics
No merged PRs in 30d

Description

_I know the [downsampling tasks examples](https://github.com/influxdata/community-templates/blob/master/downsampling/all_inputs/downsampling_tasks.yml), but I don't won't to change the downsampling tasks and contained hardcoded filters everytime we adding new data to our InfluxDB. More Information below:_

**What I want**
I want to downsample all of my data (with mean) from my raw bucket "telegraf" (frequency = 30s, retention 30 days) like this:
* telegraf -> telegraf_90d (frequency = 1h, retention 90 days)
* telegraf_90d -> telegraf_365d (frequency = 12h, retention 365 days)
..

**What I get / Error**
Unfortunately I get following error:
`could not execute task run: unsupported input type for mean aggregate: string`

**What is the cause?**
I have some collectors like [NetApp Harvest](https://github.com/NetApp/harvest) that unfortunately writes sometimes strings or booleans in "_value" and that's why I get the error above.

**What I want to prevent / What is my goal**
* **I want a working downsampling task that works with all supported data types without specifying them hardcoded.**
* I don't want to exclude all the _measurements or _fields that have strings in "_value".

**Acceptable Workaround (when we don't find any solution)**
Exclude all other data types that not contains a numeric value in "_value" from downsampling.
I thought as a QuickFix exclude all non numeric data with regex, but this won't work and I don't get any help (https://github.com/influxdata/flux/issues/3804)

-> But I would be very happy If we find a solution for my issue and not a workaround.

**My downsampling task (one of them)**
```
option task = {name: "task_telegraf_90d", every: 1h}

data = from(bucket: "telegraf")
|> range(start: -duration(v: int(v: task.every) * 2))
|> filter(fn: (r) =>
(r._measurement =~ /.*/))

data
|> aggregateWindow(fn: mean, every: 1h)
|> filter(fn: (r) =>
(exists r._value))
|> to(bucket: "telegraf_90d", org: "MYORG")
```

**Additional Informations**
InfluxDB: Version 2.0.5
VM: 8 vCores & 128GB Memory
I also write the same to the [InfluxData Community](https://community.influxdata.com/t/downsampling-task-fail-with-error-unsupported-input-type-for-mean-aggregate/23632).

**This is extremely important for us!** I am happy about any help.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the downsampling task shown in the issue and the linked downsampling_tasks.yml example, then reproduce it on InfluxDB 2.0.5 using mixed numeric, string, and boolean _value data. Determine whether a general task can downsample supported types without hardcoded filters; done means the task runs without the mean aggregate error, or the supported workaround is documented.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.