influxdata / influxdata/influxdb

Add tracing spans to last value cache

Open
#26,077 0 comments 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### Problem statement

There has been variable latency observed when using the last cache under higher concurrent query loads. There are no tracing spans to indicate where in the scan time is being taken, and contributing to that latency, in the last cache.

### Proposed solution

Extract the span context from the DataFusion `Session` that is provided to the `TableProvider::scan` implementation for the `last_cache`: https://github.com/influxdata/influxdb/blob/d8efdb4024749e3d2f05cb8b45e0b721f7cce951/influxdb3_cache/src/last_cache/table_function.rs#L71-L77

It would be worth wrapping a span around the following to generate traces:
* Filter predicate conversion: https://github.com/influxdata/influxdb/blob/d8efdb4024749e3d2f05cb8b45e0b721f7cce951/influxdb3_cache/src/last_cache/table_function.rs#L84-L89
* Record batch conversion: https://github.com/influxdata/influxdb/blob/d8efdb4024749e3d2f05cb8b45e0b721f7cce951/influxdb3_cache/src/last_cache/table_function.rs#L90

Within the latter, we could break down into more spans:
* Predicate evaluation of the cache key hierarchy: https://github.com/influxdata/influxdb/blob/d8efdb4024749e3d2f05cb8b45e0b721f7cce951/influxdb3_cache/src/last_cache/cache.rs#L427-L458
* Scanning the filtered leaf nodes: https://github.com/influxdata/influxdb/blob/d8efdb4024749e3d2f05cb8b45e0b721f7cce951/influxdb3_cache/src/last_cache/cache.rs#L460-L463

### Additional context

This is related to https://github.com/influxdata/influxdb/issues/25562

Contributor guide

Open the contributing guide

Research direction

Start in influxdb3_cache/src/last_cache/table_function.rs at the TableProvider::scan implementation, then read the filter predicate and record batch conversion sections. Follow the related cache.rs sections for cache-key predicate evaluation and filtered leaf scanning. Done means the Session span context is used and tracing spans cover those operations, including the proposed breakdown where useful.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.