influxdata / influxdata/influxdb

Add support for caching more complex predicates in TSI cache.

Open
#13,191 5 comments 0 reactions 0 assignees View on GitHub
area/performance area/tsi
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

The TSI index has a cache that provides an in-memory reverse index from:

```
(measurement, tag_key, tag_value) ==> set_of_series_ids
```

So, given a measurement name (e.g., `cpu`), tag key (e.g., `region`) and tag value (e.g., `west`), the index can very quickly return a compressed bitset of all of the series ids matching that.

This makes TSI planning faster because all of the intersection, difference and union operations, when processing more complex predicates, are done on bitsets of integers.

However, in 2.0 we use measurement name for the organisation and bucket id, so that we can support multiple buckets within the same index. Typically then predicates are always going to require intersecting two bitsets, that satisfy:

```
"_measurement" = 'cpu' AND "region" = 'west'
```

We should consider extending the functionality of the cache to support tuples like this:

```
(measurement, _measurement, tag_key, tag_value) ==> set_of_series_ids
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the TSI index cache described in the issue and reviewing how it currently indexes (measurement, tag_key, tag_value) predicates. Done means the cache supports the proposed (measurement, _measurement, tag_key, tag_value) tuple and returns the corresponding series-id bitsets for predicates spanning multiple buckets.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.