datafusion-contrib / datafusion-contrib/liquid-cache

Cache tracing instrumentation

Open
#199 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

I discussed with @nikhilanayak about tracing the cache operations yesterday. I think we are close to have a super cool cache trace analysis.
But before that, we probably need to improve how we instrument the tracing to get better information from runtime behaviors.

This is the tracer struct: https://github.com/XiangpengHao/liquid-cache/blob/0cf64a643a780fec899cc5e3c3abf71c5192a8ca/src/liquid_parquet/src/cache/tracer.rs#L24-L27

Here're the fields we currently collects:
https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/tracer.rs#L79-L86

Here is where tracing is invoked: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/store.rs#L267

Considering that the tracer is currently at the cache store level, it doesn't have the information of whether a get operation is from building filter or building output. We probably want to move the tracer to one level higher to get more semantics.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the tracer definition in src/liquid_parquet/src/cache/tracer.rs and the tracing call in src/liquid_parquet/src/cache/store.rs. Trace how cache gets are initiated so the instrumentation can distinguish filter-building from output-building operations; done means the runtime trace contains that semantic context for cache analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.