Cache FileStatus in Footer to reduce redundant NameNode RPC calls
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
### Describe the enhancement requested
Optimization reduces redundant getFileStatus() RPC calls to the NameNode
when processing Parquet files.
Benefits:
- Reduces NameNode RPC calls by 1 per Parquet file during split generation
- For workloads with thousands of small files, this can significantly reduce
NameNode pressure and improve job startup time
- Maintains full backward compatibility with existing code paths
### Component(s)
Core
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing getFileStatus() calls during Parquet split generation and inspect how the Footer is processed for each Parquet file. Confirm where a FileStatus cache can remain within the existing code path without changing behavior. Done means one fewer NameNode RPC per file, preserved backward compatibility, and reduced calls for workloads with many small files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100