apache / apache/datafusion

Caching datasets in context [Question]

Open
#1,396 10 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Hi,
Is there a way to somehow cache a large CSV or Parquet file that was loaded to CTX in order to avoid reading it again each time when a new process tried to access it
What is the proper way to manage this in case you want to write something like a server that needs to answer multiple requests about the same file?

Thanks

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. First clarify whether the request concerns caching CSV or Parquet data loaded into CTX across processes, and define the server's expected reuse and invalidation behavior before identifying the relevant DataFusion context and dataset-loading code.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.