apache / apache/datafusion-comet
Share Parquet footer and page index metadata across tasks in an executor
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
Comet's native Parquet scan caches each file's footer and page index in DataFusion's `FileMetadataCache`, which it obtains from the task's `RuntimeEnv` (`native/core/src/parquet/parquet_exec.rs`). Comet builds a fresh `RuntimeEnv` for every Spark task, so that cache is discarded when the task ends.
Two tasks on the same executor therefore re-fetch and re-parse the same footer, once each. This recurs in two common shapes:
- a large Parquet file split into many Spark partitions, each read by a different task;
- the same table scanned again by a later stage or query in the same application.
Each miss costs a round trip to object storage plus a Thrift parse. Spark's own reader has the same per-task behavior, so this is parity with Spark, but Velox caches file handles and footers process-wide, and this is one of the gaps identified as item P5 in #4842.
Sharing the cache also reduces memory: today each concurrent task gets its own 50MB budget.
## Describe the potential solution
Hold the metadata caches in a process-wide registry keyed by object store URL, and use it at the scan's single call site, falling back to the per-task cache when sharing is disabled.
This requires the cache validity check to actually work first. DataFusion validates cached metadata against `(size, last_modified)`, but Comet builds its `PartitionedFile` with `PartitionedFile::new_with_range`, which hardcodes `last_modified` to epoch 0, and `SparkPartitionedFile` carries no modification time. Validation is therefore size-only today. That is harmless for a cache that lives for one task and a wrong-results hazard for one that lives for the executor, so Spark's `PartitionedFile.modificationTime` needs to reach `object_meta.last_modified`.
## Additional context
Item P5 of #4842.
Contributor guide
Research direction
Start in native/core/src/parquet/parquet_exec.rs and trace the scan's single metadata-cache call site, then follow how SparkPartitionedFile supplies file metadata. First make the cache validity inputs include modification time before assessing the process-wide registry keyed by object store URL. Done means executor tasks can share valid footer and page-index metadata while sharing can still be disabled and per-task caching remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100