aws-samples / aws-samples/spark-streaming-sql-s3-connector
[bug] AvailableNow trigger causes stream to stop after one batch
- Dominant language
- Scala
- Stars
- 16
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
The availableNow trigger appears to complete after a single batch of SQS messages is consumed, even if there are plenty of messages on the queue:
```
df = (spark.readStream.format("s3-connector")
.option("spark.s3conn.queueRegion": "us-east-1")
.option("spark.s3conn.fileFormat": "parquet")
.option("spark.s3conn.maxFilesPerTrigger": "2048")
.option("spark.s3conn.queueUrl": "https://sqs.us-east-1.amazonaws.com//")
.option("spark.s3conn.queueFetchWaitTimeoutSeconds": "10")
.option("spark.s3conn.sqs.longPollingWaitTimeSeconds": "5")
.option("spark.s3conn.sqs.visibilityTimeoutSeconds": "600")
.option("spark.s3conn.pathGlobFilter": "*.parquet.gz")
.schema(schema)
.readStream("s3:...."))
df.writeStream.trigger(availableNow=True).foreachBatch(stream_callback)
.option("checkpointLocation", )
```
This was reproduced on Databricks DBR 14.3
Contributor guide
Research direction
Reproduce the supplied SQS setup on Databricks DBR 14.3 and inspect the connector path exercised by readStream with trigger(availableNow=True). There is no source file or test named in the issue; done means queued SQS batches continue to be consumed until the queue is exhausted rather than stopping after one batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, scala, spark
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100