Make S3A readahead option configurable for compactions
Open
compactions-module
enhancement
ingest-module
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Background
The property `fs.s3a.readahead.range` on the Hadoop configuration we use for reading / writing Parquet files is configurable for queries. It is not configurable for compactions or for standard ingest.
### Description
We'd like to be able to configure `fs.s3a.readahead.range` for compactions and standard ingest.
### Analysis
We need to add an instance property to set the above property for compactions and standard ingest. We can use this to set`fs.s3a.readahead.range` in `getConfigurationForECS` in `HadoopConfigurationProvider`.
Contributor guide
Assessment
This issue has not been assessed yet.