apache / apache/hudi

Cloud incremental sources cannot select more than one file extension

Open
#19,671 1 comment 0 reactions 1 assignee Claimed by @rahil-c View on GitHub
area:config area:ingest priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` can only select one file
extension per pipeline. `hoodie.streamer.source.cloud.data.select.file.extension` is
interpolated into a single SQL predicate, so a bucket holding more than one file type
cannot be ingested by one streamer.

Setting a list looks like it should work, and fails quietly. With
`select.file.extension=json,jsonl` the generated filter becomes:

```
s3.object.size > 0 and s3.object.key like '%json,jsonl'
```

That is a literal match on the comma joined string, so it matches nothing. The job
succeeds, commits nothing, and reports no error, which is hard to diagnose because
there is no signal that the filter is at fault.

### Expected

A comma separated value should select files matching any one of the extensions, while a
single value keeps behaving exactly as it does today.

### Notes

The same config also falls back to the data file format when unset, which defaults to
`parquet`. That behaviour is long standing and remains unchanged here, but it is
currently undocumented and worth pinning with a test.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.