apache / apache/hudi

S3 incremental source should impose limit on num of files consumed per batch

Open
#18,075 0 comments 0 reactions 0 assignees View on GitHub
type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
Currently the cloud incremental source configures limit on number of bytes read from source. But with very small files, the number of files read can increase drastically and managing all the files metadata in driver can lead to OOM or memory contention. The Issue aims to add a limit on the number of rows read by the source as well to reduce the memory overhead on driver.

**What you expected:**

**Steps to reproduce:**
1.
2.
3.

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the S3 incremental source and inspect how its existing bytes-read limit is configured and enforced. Determine the appropriate file-count limit and verify that batches stop before excessive small-file metadata is accumulated, while preserving the existing source behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.