datafusion-contrib / datafusion-contrib/liquid-cache
Cache tracing instrumentation
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
I discussed with @nikhilanayak about tracing the cache operations yesterday. I think we are close to have a super cool cache trace analysis.
But before that, we probably need to improve how we instrument the tracing to get better information from runtime behaviors.
This is the tracer struct: https://github.com/XiangpengHao/liquid-cache/blob/0cf64a643a780fec899cc5e3c3abf71c5192a8ca/src/liquid_parquet/src/cache/tracer.rs#L24-L27
Here're the fields we currently collects:
https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/tracer.rs#L79-L86
Here is where tracing is invoked: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/store.rs#L267
Considering that the tracer is currently at the cache store level, it doesn't have the information of whether a get operation is from building filter or building output. We probably want to move the tracer to one level higher to get more semantics.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tracer definition in src/liquid_parquet/src/cache/tracer.rs and the tracing call in src/liquid_parquet/src/cache/store.rs. Trace how cache gets are initiated so the instrumentation can distinguish filter-building from output-building operations; done means the runtime trace contains that semantic context for cache analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100