influxdata / influxdata/kapacitor
Dynamic field names in sideload()?
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have a environment that we are monitoring with a pretty high number of filesystems that presents a lot of variability in required threshold values, I've been using sideload with success for having host or group specific thresholds, but I am not sure I am going to be able to achieve my desired goal with device level overrides.
Here is what I have so far:
```
var window = 5m
var data = stream
|from()
.measurement('disk')
.where(lambda: "host" == 'test')
.groupBy('host', 'device')
|window()
.period(window)
|sideload()
.source('file:///etc/kapacitor/sideload')
.order('host/{{.host}}.yml', 'serverclass/{{.serverclass}}.yml')
.field('disk_usage_warning_level', 95)
.field('disk_usage_critical_level', 97)
|alert()
.id('{{ .TaskName }}{{ index .Tags "host"}}//{{ index .Tags "device" }}')
.message('[{{ .TaskName }}] {{ index .Tags "host" }} disk: {{ index .Tags "device" }} is at {{ index .Fields "used_percent" | printf "%0.2f" }}% utilization.')
.warn(lambda: "used_percent" > "disk_usage_warning_level")
.crit(lambda: "used_percent" > "disk_usage_critical_level")
.pagerDuty2()
```
This works great, but it doesn't allow me to override at a path level.
Right now my 'test.yaml' override file for the test machine looks like:
```
#test.yaml
disk_usage_warning_level: 90
disk_usage_critical_level: 95
```
I want to be able to do something like this:
```
#test.yaml
disk_usage_warning:
/var/local/docstore/index/docserviceD/rev2-complete: 30 #Matches a specific mountpoint
/var.*: 99 #Regex matches any other mountpoints in /var
default: 90 #Uses this if nothing more specific is found
disk_usage_critical: 95
```
Which I'm fairly certain is currently impossible.
I know normally I would be able to do something like adding to sideload.order:
'host/{{.host}}/{{.path}}.yaml', but the problem is that my paths have backslashes in them (of course) so I can't create a filename out of them anyways (plus, our environment has over 10,000 individual file systems....that is a lot of yaml files to manage).
Anyone have any suggestions on a better way to do this? Is it possible to dynamically name the sideload field to include the name of the path or something like that?
Contributor guide
Research direction
Start by reviewing the sideload() configuration, especially its order() and field() entries, alongside the test.yaml examples in the issue. Determine whether path-level matching, regex fallbacks, defaults, or dynamically named fields are supported; done means defining a documented approach for applying per-filesystem thresholds without one YAML file per path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100