influxdata / influxdata/kapacitor
[feature request] Support bulk operations on fields
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Add ability to support bulk operations on field values without actually specifying the field key. This will be very helpful when using Kapacitor as a downsampling engine (similar to the use-case discussed in the documentation [here](https://docs.influxdata.com/kapacitor/v1.2/examples/continuous_queries/)). From my perspective, supporting this effectively requires:
* Ability to pass wildcard operators to InfluxQL functions (`|mean(*)`)
* Ability to reference the anonymous field key. For example, once a wildcard operator is used, I need to be able to reference the name of the field (referenced as `$fieldkey` below) within the stream for naming.
More concrete examples:
#### Numeric Data
Aggregating numeric data is fairly straightforward. For example, I would like to be able to apply mathematical operations to anonymous fields based on their type. This would be similar to running the query in InfluxQL:
```
SELECT sum(*), min(*), max(*), count(*) FROM /.*/ WHERE time > now() - 1m
```
Where InfluxQL automatically applies the operations to all numeric fields (even if string or boolean fields are present). In TICKscript, I could see this looking similar to:
```
var data = stream
|from().groupBy(*)
|window().every(1m).period(1m)
var count = data
// Rename 'count' result to be the same name as the original field key for all fields
|count(*)
.as($fieldkey)
var sum = data
|sum(*)
...
```
Where the InfluxQL functions accept a wildcard operator `*`. When the wildcard is used, type errors would be ignored (for example, attempting to use `sum()` on string types), or a property method would need to be set to enable ignoring type errors. Once a wildcard is referenced, you can then reference the field key with the `$fieldkey` operator.
#### Non-numeric Data
This enhancement could also leverage the type checks proposed in https://github.com/influxdata/kapacitor/issues/1201, which would allow the user to specify different actions to take based on the type of the field. One fairly common request I hear is how to effectively downsample string data. For example, based on the input data:
```
measurement,tagkey=tagvalue mystring="test1"
measurement,tagkey=tagvalue mystring="test2"
measurement,tagkey=tagvalue mystring="test2"
measurement,tagkey=tagvalue mystring="test1"
measurement,tagkey=tagvalue mystring="test1"
```
Taking the aggregate of this data will look like:
```
measurement,tagkey=tagvalue mystring_count=3i,mystring="test1"
measurement,tagkey=tagvalue mystring_count=2i,mystring="test2"
```
Doing this in TICKscript currently requires that you know the field keys ahead of time (which is not always possible). Being able to generalize this algorithm into something like:
```
var data = stream
|from().groupBy(*)
|window().every(1m).period(1m)
data
|where(lambda: isinstance(*, string))
// convert string field to tag
|eval(lambda: "$fieldkey")
.keep()
.tags($fieldkey)
// group by new tag
|groupBy(*)
// count number of fields per tag
|count(*)
.as($fieldkey + '_count')
// remove created tag
|delete().tag($fieldkey)
```
Would be amazingly helpful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading Kapacitor's handling of InfluxQL functions and TICKscript field references, then review the related type-checking proposal in issue #1201. Done means wildcard operations can apply across compatible fields, expose each field key for naming, and support the numeric and string downsampling examples described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100