Caching datasets in context [Question]
Open
question
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Hi,
Is there a way to somehow cache a large CSV or Parquet file that was loaded to CTX in order to avoid reading it again each time when a new process tried to access it
What is the proper way to manage this in case you want to write something like a server that needs to answer multiple requests about the same file?
Thanks
Contributor guide
Research direction
No source file, test, or entry point is named. First clarify whether the request concerns caching CSV or Parquet data loaded into CTX across processes, and define the server's expected reuse and invalidation behavior before identifying the relevant DataFusion context and dataset-loading code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100