apache / apache/parquet-java

Cache FileStatus in Footer to reduce redundant NameNode RPC calls

Open
#3,394 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

### Describe the enhancement requested

Optimization reduces redundant getFileStatus() RPC calls to the NameNode
when processing Parquet files.

Benefits:
- Reduces NameNode RPC calls by 1 per Parquet file during split generation
- For workloads with thousands of small files, this can significantly reduce
  NameNode pressure and improve job startup time
- Maintains full backward compatibility with existing code paths

### Component(s)

Core

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing getFileStatus() calls during Parquet split generation and inspect how the Footer is processed for each Parquet file. Confirm where a FileStatus cache can remain within the existing code path without changing behavior. Done means one fewer NameNode RPC per file, preserved backward compatibility, and reduced calls for workloads with many small files.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.