influxdata / influxdata/community-templates
Downsampling Task that works on all data types without hardcoded filters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 376
- Forks
- 163
- PR merge metrics
- No merged PRs in 30d
Description
_I know the [downsampling tasks examples](https://github.com/influxdata/community-templates/blob/master/downsampling/all_inputs/downsampling_tasks.yml), but I don't won't to change the downsampling tasks and contained hardcoded filters everytime we adding new data to our InfluxDB. More Information below:_
**What I want**
I want to downsample all of my data (with mean) from my raw bucket "telegraf" (frequency = 30s, retention 30 days) like this:
* telegraf -> telegraf_90d (frequency = 1h, retention 90 days)
* telegraf_90d -> telegraf_365d (frequency = 12h, retention 365 days)
..
**What I get / Error**
Unfortunately I get following error:
`could not execute task run: unsupported input type for mean aggregate: string`
**What is the cause?**
I have some collectors like [NetApp Harvest](https://github.com/NetApp/harvest) that unfortunately writes sometimes strings or booleans in "_value" and that's why I get the error above.
**What I want to prevent / What is my goal**
* **I want a working downsampling task that works with all supported data types without specifying them hardcoded.**
* I don't want to exclude all the _measurements or _fields that have strings in "_value".
**Acceptable Workaround (when we don't find any solution)**
Exclude all other data types that not contains a numeric value in "_value" from downsampling.
I thought as a QuickFix exclude all non numeric data with regex, but this won't work and I don't get any help (https://github.com/influxdata/flux/issues/3804)
-> But I would be very happy If we find a solution for my issue and not a workaround.
**My downsampling task (one of them)**
```
option task = {name: "task_telegraf_90d", every: 1h}
data = from(bucket: "telegraf")
|> range(start: -duration(v: int(v: task.every) * 2))
|> filter(fn: (r) =>
(r._measurement =~ /.*/))
data
|> aggregateWindow(fn: mean, every: 1h)
|> filter(fn: (r) =>
(exists r._value))
|> to(bucket: "telegraf_90d", org: "MYORG")
```
**Additional Informations**
InfluxDB: Version 2.0.5
VM: 8 vCores & 128GB Memory
I also write the same to the [InfluxData Community](https://community.influxdata.com/t/downsampling-task-fail-with-error-unsupported-input-type-for-mean-aggregate/23632).
**This is extremely important for us!** I am happy about any help.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the downsampling task shown in the issue and the linked downsampling_tasks.yml example, then reproduce it on InfluxDB 2.0.5 using mixed numeric, string, and boolean _value data. Determine whether a general task can downsample supported types without hardcoded filters; done means the task runs without the mean aggregate error, or the supported workaround is documented.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100