[SUPPORT] DeltaStreamer non-continuous behavior for S3EventsSource + S3EventsHoodieIncrSource
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Issues**
I'm finding a bit of trouble in double-checking if this is the expected behavior or even if I'm missing parametrizations for non-continuous pipelines (run-once) with both S3EventsSource + IncrSource
The setup is the same as the [documentation](https://hudi.apache.org/blog/2021/08/23/s3-events-source/) suggests, with the removal of the `--continuous` flag.
`S3EventsSource` - SQS
1. If we have an `S3EventsSource` for an SQS queue with a small `hoodie.deltastreamer.s3.source.queue.max.messages.per.batch`- it runs once and then stops, but it only runs for a single batch. Is there a way to make DeltaStreamer run for the entire available queue, without having to set an enormous `max.messages.per.batch`?
`S3EventsHoodieIncrSource`
1. For a first run, if the `hoodie.deltastreamer.source.hoodieincr.read_latest_on_missing_ckpt` is set to `true` - then it did read the last **changes** committed in the source event table, but not the entire metadata table. This is a bit confusing as the source (S3EventsSource) job might have run more times than the downstream S3EventsHoodieIncrSource job so it failed to capture all events/s3 objects registered in the metadata source table. What is the expected behavior there?
2. Documentation mentions a fan-out behavior for the queue, the suggested way to filter bucket paths/partitions here is by using [Transformers](https://hudi.apache.org/docs/next/transforms/)? (so it would be possible to have a fan-out with a single queue/S3EventsSource job)
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.3.1
* Hive version : 3.1.3
* Hadoop version : 3.3.3
* SQS JAR version: 1.12.22
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the S3 events source documentation linked in the issue and the S3EventsSource and S3EventsHoodieIncrSource entry points. Reproduce the run-once behavior with the stated SQS and checkpoint settings, then determine and document the expected queue consumption, incremental-read, and fan-out behavior; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100