influxdata / influxdata/kapacitor
Kapacitor not respecting group by in tick script
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We are using Kapacitor version:
`Kapacitor OSS 1.5.2 (git: HEAD 3086452d00830e01d932838d8c6d1df818648ad3)`
We have a Tick Script defined as:
```
stream
|from()
.database('prodmondb')
.retentionPolicy('default')
.measurement('as-join-cluster')
.groupBy('aerospike_host','region')
.where(lambda: isPresent("memory_free_pct"))
|window()
.period(1m)
.every(1m)
.align()
|min('memory_free_pct')
.as('memory_free_pct')
|log()
|influxDBOut()
.database('prodaggdb')
.retentionPolicy('default')
.measurement('agg_as-join-cluster')
.precision('s')
```
The log statement, and the corresponding Influx write, contain many tags that we believe we should be omiting due to our group by including only two tags:
```
ts=2019-01-21T15:54:00.848-05:00 lvl=info msg=point service=kapacitor task_master=main task=stream_cq_aerospike_agg_as-join-cluster_memory_free_pct.tick node=log4 prefix= name=as-join-cluster db= rp= gr
oup=aerospike_host=as-join-cluster:3000,region=ap-northeast-1 dimension_0=aerospike_host dimension_1=region tag_service=telegraf-hm tag_aerospike_host=as-join-cluster:3000 tag_host=hm-ip-10-10-10-10 tag_namespace=joins tag_node_name=BB2 tag_region=ap-northeast-1 field_client_delete_success=0 field_client_write_timeou
t=35 field_{joins}_udf_hist_track_slice=10 field_tomb_raider_eligible_age=86400 field_client_udf_timeout=0 field_enable_benchmarks_udf=false field_migrate_record_receives=50690075 field_high_water_memor
y_pct=60 field_migrate_records_transmitted=184104849 field_query_agg_avg_rec_count=0 field_storage_engine.min_avail_pct=5 field_query_reqs=0 field_retransmit_udf_sub_repl_write=0 field_storage_engine=de
vice field_client_read_not_found=59020891 field_query_short_queue_full=0 field_migrate_tx_partitions_active=0 field_udf_sub_udf_timeout=0 field_migrate_signals_active=0 field_udf_sub_lang_read_success=0
field_disallow_null_setname=false field_migrate_tx_partitions_remaining=0 field_batch_sub_proxy_timeout=0 field_batch_sub_tsvc_timeout=0 field_enable_benchmarks_batch_sub=false field_retransmit_client_
udf_repl_write=0 field_evicted_objects=9596941549 field_prole_objects=12166102 field_rack_id=0 field_storage_engine.post_write_queue=256 field_non_replica_tombstones=0 field_fail_key_busy=0 field_{joins
}_query_hist_track_slice=10 field_hwm_breached=false field_query_fail=0 field_{joins}_read_hist_track_thresholds=1,8,64 field_obj_size_hist_max=100 field_client_udf_complete=0 field_migrate_sleep=1 fiel
d_migrate_record_retransmits=0 field_high_water_disk_pct=50 field_data_in_index=false field_client_write_success=37109914538 field_retransmit_client_write_dup_res=0 field_retransmit_client_udf_dup_res=0
field_query_udf_bg_failure=0 field_udf_sub_udf_error=0 field_ldt_enabled=false field_evict_tenths_pct=5 field_query_agg_success=0 field_enable_hist_proxy=false field_cold_start_evict_ttl=4294967295 fie
ld_ns_forward_xdr_writes=false field_migrate_records_skipped=50651412 field_non_replica_objects=0 field_client_proxy_error=0 field_geo_region_query_points=0 field_enable_xdr=false field_repl_factor=2 fi
eld_migrate_order=5 field_enable_benchmarks_write=false field_storage_engine.enable_benchmarks_storage=false field_migrate_tx_partitions_imbalance=0 field_tombstones=0 field_storage_engine.cold_start_em
pty=false field_geo2dsphere_within.earth_radius_meters=6371000 field_migrate_signals_remaining=0 field_memory_used_index_bytes=1497305792 field_xdr_write_error=0 field_max_ttl=315360000 field_current_ti
me=285799980 field_storage_engine.fsync_max_sec=0 field_retransmit_nsup_repl_write=0 field_memory_size=16106127360 field_scan_aggr_error=0 field_memory_free_pct=90 field_query_agg=0 field_scan_aggr_comp
lete=0 field_device_total_bytes=474999685120 field_evict_hist_buckets=10000 field_client_read_success=2577590541 field_storage_engine.write_block_size=1048576 field_evict_ttl=3 field_storage_engine.data
_in_memory=false field_scan_basic_abort=0 field_client_delete_error=0 field_migrate_tx_partitions_initial=282 field_partition_tree_locks=8 field_{joins}_query_hist_track_back=300 field_query_long_queue_
full=0 field_client_delete_timeout=0 field_prole_sub_objects=0 field_retransmit_batch_sub_dup_res=0 field_{joins}_write_hist_track_back=300 field_storage_engine.defrag_queue_min=0 field_client_write_err
or=934450874 field_storage_engine.flush_max_ms=1000 field_master_tombstones=0 field_retransmit_client_delete_dup_res=0 field_client_lang_error=0 field_scan_udf_bg_complete=0 field_migrate_rx_partitions_
remaining=0 field_client_lang_read_success=0 field_enable_benchmarks_read=false field_geo2dsphere_within.max_level=30 field_fail_record_too_big=0 field_master_objects=11229300 field_scan_udf_bg_abort=0
field_memory_used_bytes=1497305792 field_migrate_rx_instances=0 field_device_free_pct=85 field_batch_sub_read_not_found=0 field_query_agg_abort=0 field_query_lookup_abort=0 field_single_bin=false field_
nsup_cycle_sleep_pct=63 field_udf_sub_lang_write_success=0 field_batch_sub_proxy_error=0 field_memory_used_sindex_bytes=0 field_client_lang_delete_success=0 field_truncate_lut=0 field_device_used_bytes=
70091048704 field_deleted_last_bin=0 field_xdr_write_timeout=0 field_scan_basic_error=0 field_query_lookup_avg_rec_count=0 field_migrate_rx_partitions_active=0 field_write_commit_level_override=off fiel
d_{joins}_write_hist_track_thresholds=1,8,64 field_expired_objects=18192665829 field_prole_tombstones=0 field_stop_writes_pct=90 field_batch_sub_read_timeout=0 field_tomb_raider_period=86400 field_stora
ge_engine.defrag_lwm_pct=50 field_geo2dsphere_within.level_mod=1 field_batch_sub_tsvc_error=0 field_udf_sub_udf_complete=0 field_storage_engine.device=/dev/nvme0n1 field_geo_region_query_falsepos=0 fiel
d_device_available_pct=83 field_nsup_cycle_duration=20 field_geo2dsphere_within.min_level=1 field_allow_xdr_writes=true field_udf_sub_tsvc_error=0 field_client_tsvc_timeout=0 field_read_consistency_leve
l_override=off field_query_lookups=0 field_scan_aggr_abort=0 field_{joins}_read_hist_track_back=300 field_geo_region_query_cells=0 field_batch_sub_proxy_complete=0 field_ldt_gc_rate=0 field_query_long_r
eqs=0 field_batch_sub_read_error=0 field_{joins}_write_hist_track_slice=10 field_conflict_resolution_policy=generation field_fail_xdr_forbidden=0 field_retransmit_client_delete_repl_write=0 field_client
_lang_write_success=0 field_storage_engine.defrag_sleep=1000 field_{joins}_query_hist_track_thresholds=1,8,64 field_udf_sub_tsvc_timeout=0 field_enable_benchmarks_udf_sub=false field_storage_engine.writ
e_threads=1 field_client_proxy_timeout=0 field_udf_sub_lang_error=0 field_query_lookup_success=0 field_sub_objects=0 field_retransmit_client_write_repl_write=1569 field_{joins}_udf_hist_track_back=300 f
ield_xdr_write_success=0 field_retransmit_client_read_dup_res=0 field_query_agg_error=0 field_sets_enable_xdr=true field_retransmit_udf_sub_dup_res=0 field_cache_read_pct=95 field_ldt_page_size=8192 fie
ld_fail_generation=0 field_storage_engine.max_write_cache=536870912 field_client_delete_not_found=0 field_stop_writes=false field_storage_engine.defrag_startup_minimum=10 field_default_ttl=86400 field_c
lient_proxy_complete=2490 field_master_sub_objects=0 field_truncated_records=13365185689 field_allow_nonxdr_writes=true field_client_read_error=0 field_query_lookup_error=0 field_scan_udf_bg_error=0 fie
ld_batch_sub_read_success=0 field_storage_engine.scheduler_mode=null field_storage_engine.enable_osync=false field_client_read_timeout=0 field_client_udf_error=0 field_migrate_tx_instances=0 field_objec
ts=23395402 field_sindex.num_partitions=32 field_geo2dsphere_within.strict=true field_non_expirable_objects=0 field_{joins}_read_hist_track_slice=10 field_storage_engine.disable_odirect=false field_clie
nt_tsvc_error=0 field_migrate_rx_partitions_initial=117 field_non_replica_sub_objects=0 field_memory_used_data_bytes=0 field_xmem_id=1 field_storage_engine.filesize=17179869184 field_geo_region_query_re
qs=0 field_available_bin_names=32751 field_{joins}_udf_hist_track_thresholds=1,8,64 field_migrate_retransmit_ms=5000 field_udf_sub_lang_delete_success=0 field_query_udf_bg_success=0 field_scan_basic_com
plete=0 field_query_short_reqs=0 field_geo2dsphere_within.max_cells=12 field_storage_engine.tomb_raider_sleep=1000 field_partition_tree_sprigs=64 time=2019-01-21T20:54:00Z
```
You can see the tags for the source db:
```
name: as-join-cluster
tagKey
------
aerospike_host
host
namespace
node_name
region
service
```
Since the group by of the Tick only has `aerospike_host` and `region`, we expect only those tags in the destination db but we get all of them:
```
name: agg_as-join-cluster
tagKey
------
aerospike_host
host
namespace
node_name
region
service
```
We have many Tickscripts with similar format as the above. We're seeing this issue on the Ticks that use `min`, `max`, and `percentile` on aggregation. We are not seeing this issue with the Ticks that use `sum` for aggreagtion.
Happy to provide any additional info. Thanks!
Contributor guide
Research direction
Reproduce the Tick Script on Kapacitor OSS 1.5.2 using the shown from, groupBy, window, min, and influxDBOut pipeline, then compare its tags with an equivalent sum pipeline. Trace how min, max, and percentile aggregation outputs preserve group tags and verify the destination measurement contains only aerospike_host and region.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100