influxdata / influxdata/influxdb

Benchmark performance of slow path for tagKeys and tagValues

Open
#19,871 0 comments 0 reactions 0 assignees View on GitHub
team/query
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

As part of these issues:
https://github.com/influxdata/influxdb/issues/19794
https://github.com/influxdata/influxdb/issues/19806

We added a "slow path" that does block scans in order to discover with field tag key/value combinations exist. For most cases, we can still use the index and shard metadata to answer metaqueries.

However, using the index and shard metadata will not provide answers that respect the time range passed into `tagKeys` and `tagValues`. It may be that we need to always perform the block scan in order to provide a correct answer. The work to do that is this issue:
https://github.com/influxdata/influxdb/issues/19814

But how slow is the slow path? Is it prohibitively slow to do it for every metaquery? We need a benchmark to help us answer this question.

So the work of this issue is:
- Create a database that is representative of a realistic scenario. Some research will be needed to determine what this means: how many series, how many fields, how many tag keys and values. Perhaps the product team could provide some input here.
- Measure performance of `tagKeys` and `tagValues` with an emphasis on how they tend to be invoked in the UI's Query Builder. To understand how the Query Builder works, I've found it useful to use a debugger to look at the metaqueries that are submitted when using the UI.

DOD:
- A reproducible benchmark exists and is merged to master
- We have some useful data on the performance of metaqueries

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issues 19794, 19806, and 19814, then inspect the existing slow path and use a debugger to observe the UI Query Builder's tagKeys and tagValues metaqueries. Create a realistic database, measure those queries, and add a reproducible benchmark. Done means the benchmark is merged to master and provides useful performance data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.