Cloud incremental sources cannot select more than one file extension
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` can only select one file
extension per pipeline. `hoodie.streamer.source.cloud.data.select.file.extension` is
interpolated into a single SQL predicate, so a bucket holding more than one file type
cannot be ingested by one streamer.
Setting a list looks like it should work, and fails quietly. With
`select.file.extension=json,jsonl` the generated filter becomes:
```
s3.object.size > 0 and s3.object.key like '%json,jsonl'
```
That is a literal match on the comma joined string, so it matches nothing. The job
succeeds, commits nothing, and reports no error, which is hard to diagnose because
there is no signal that the filter is at fault.
### Expected
A comma separated value should select files matching any one of the extensions, while a
single value keeps behaving exactly as it does today.
### Notes
The same config also falls back to the data file format when unset, which defaults to
`parquet`. That behaviour is long standing and remains unchanged here, but it is
currently undocumented and worth pinning with a test.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.