apache / apache/hudi

[SUPPORT] DeltaStreamer non-continuous behavior for S3EventsSource + S3EventsHoodieIncrSource

Open
#9,398 2 comments 0 reactions 0 assignees View on GitHub
area:aws area:ingest priority:high status:triaged type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Issues**

I'm finding a bit of trouble in double-checking if this is the expected behavior or even if I'm missing parametrizations for non-continuous pipelines (run-once) with both S3EventsSource + IncrSource

The setup is the same as the [documentation](https://hudi.apache.org/blog/2021/08/23/s3-events-source/) suggests, with the removal of the `--continuous` flag.

`S3EventsSource` - SQS
1. If we have an `S3EventsSource` for an SQS queue with a small `hoodie.deltastreamer.s3.source.queue.max.messages.per.batch`- it runs once and then stops, but it only runs for a single batch. Is there a way to make DeltaStreamer run for the entire available queue, without having to set an enormous `max.messages.per.batch`?

`S3EventsHoodieIncrSource`
1. For a first run, if the `hoodie.deltastreamer.source.hoodieincr.read_latest_on_missing_ckpt` is set to `true` - then it did read the last **changes** committed in the source event table, but not the entire metadata table. This is a bit confusing as the source (S3EventsSource) job might have run more times than the downstream S3EventsHoodieIncrSource job so it failed to capture all events/s3 objects registered in the metadata source table. What is the expected behavior there?
2. Documentation mentions a fan-out behavior for the queue, the suggested way to filter bucket paths/partitions here is by using [Transformers](https://hudi.apache.org/docs/next/transforms/)? (so it would be possible to have a fan-out with a single queue/S3EventsSource job)

**Environment Description**

* Hudi version : 0.12.1

* Spark version : 3.3.1

* Hive version : 3.1.3

* Hadoop version : 3.3.3

* SQS JAR version: 1.12.22

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the S3 events source documentation linked in the issue and the S3EventsSource and S3EventsHoodieIncrSource entry points. Reproduce the run-once behavior with the stated SQS and checkpoint settings, then determine and document the expected queue consumption, incremental-read, and fan-out behavior; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
cloud, data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.