aws-samples / aws-samples/spark-streaming-sql-s3-connector

[bug] AvailableNow trigger causes stream to stop after one batch

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
16
Forks
4
PR merge metrics
No merged PRs in 30d

Description

The availableNow trigger appears to complete after a single batch of SQS messages is consumed, even if there are plenty of messages on the queue:

```
df = (spark.readStream.format("s3-connector")
.option("spark.s3conn.queueRegion": "us-east-1")
.option("spark.s3conn.fileFormat": "parquet")
.option("spark.s3conn.maxFilesPerTrigger": "2048")
.option("spark.s3conn.queueUrl": "https://sqs.us-east-1.amazonaws.com//")
.option("spark.s3conn.queueFetchWaitTimeoutSeconds": "10")
.option("spark.s3conn.sqs.longPollingWaitTimeSeconds": "5")
.option("spark.s3conn.sqs.visibilityTimeoutSeconds": "600")
.option("spark.s3conn.pathGlobFilter": "*.parquet.gz")
.schema(schema)
.readStream("s3:...."))

df.writeStream.trigger(availableNow=True).foreachBatch(stream_callback)
.option("checkpointLocation", )
```

This was reproduced on Databricks DBR 14.3

Contributor guide

Open the contributing guide

Research direction

Reproduce the supplied SQS setup on Databricks DBR 14.3 and inspect the connector path exercised by readStream with trigger(availableNow=True). There is no source file or test named in the issue; done means queued SQS batches continue to be consumed until the queue is exhausted rather than stopping after one batch.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala, spark
Domain
cloud, data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.