gchq / gchq/sleeper

Make S3A readahead option configurable for compactions

Open
#1,236 0 comments 0 reactions 0 assignees View on GitHub
compactions-module enhancement ingest-module
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Background

The property `fs.s3a.readahead.range` on the Hadoop configuration we use for reading / writing Parquet files is configurable for queries. It is not configurable for compactions or for standard ingest.

### Description

We'd like to be able to configure `fs.s3a.readahead.range` for compactions and standard ingest.

### Analysis

We need to add an instance property to set the above property for compactions and standard ingest. We can use this to set`fs.s3a.readahead.range` in `getConfigurationForECS` in `HadoopConfigurationProvider`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.