apache / apache/datafusion-comet

Share Parquet footer and page index metadata across tasks in an executor

Open
#4,994 0 comments 0 reactions 0 assignees View on GitHub
area:scan enhancement performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

Comet's native Parquet scan caches each file's footer and page index in DataFusion's `FileMetadataCache`, which it obtains from the task's `RuntimeEnv` (`native/core/src/parquet/parquet_exec.rs`). Comet builds a fresh `RuntimeEnv` for every Spark task, so that cache is discarded when the task ends.

Two tasks on the same executor therefore re-fetch and re-parse the same footer, once each. This recurs in two common shapes:

- a large Parquet file split into many Spark partitions, each read by a different task;
- the same table scanned again by a later stage or query in the same application.

Each miss costs a round trip to object storage plus a Thrift parse. Spark's own reader has the same per-task behavior, so this is parity with Spark, but Velox caches file handles and footers process-wide, and this is one of the gaps identified as item P5 in #4842.

Sharing the cache also reduces memory: today each concurrent task gets its own 50MB budget.

## Describe the potential solution

Hold the metadata caches in a process-wide registry keyed by object store URL, and use it at the scan's single call site, falling back to the per-task cache when sharing is disabled.

This requires the cache validity check to actually work first. DataFusion validates cached metadata against `(size, last_modified)`, but Comet builds its `PartitionedFile` with `PartitionedFile::new_with_range`, which hardcodes `last_modified` to epoch 0, and `SparkPartitionedFile` carries no modification time. Validation is therefore size-only today. That is harmless for a cache that lives for one task and a wrong-results hazard for one that lives for the executor, so Spark's `PartitionedFile.modificationTime` needs to reach `object_meta.last_modified`.

## Additional context

Item P5 of #4842.

Contributor guide

Open the contributing guide

Research direction

Start in native/core/src/parquet/parquet_exec.rs and trace the scan's single metadata-cache call site, then follow how SparkPartitionedFile supplies file metadata. First make the cache validity inputs include modification time before assessing the process-wide registry keyed by object store URL. Done means executor tasks can share valid footer and page-index metadata while sharing can still be disabled and per-task caching remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.