redpanda-data / redpanda-data/connect
aws_s3: configurable idle back-off for SQS polling (sqs.idle_poll_period)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Problem
The aws_s3 input, when SQS-driven, throttles empty-queue polling with a hardcoded 500ms between ReceiveMessage calls (internal/impl/aws/s3/input.go, sqsTargetReader.Pop). With short polling that is ~2 requests/s/instance on an idle queue; even with wait_time_seconds: 20 (the SQS maximum) the floor is ~1 request / 20.5s. There is no way to back off further.
For deployments that poll SQS from outside the AWS account (e.g. across a NAT gateway), this idle traffic adds avoidable request volume and NAT data-processing at fleet scale.
Proposal
Add an optional sqs.idle_poll_period duration field. When set (> 0) it replaces the 500ms throttle applied after an empty receive; the default 0s preserves current behaviour exactly. It stacks after wait_time_seconds long-polling and trades first-message latency for fewer requests — suited to batch S3 ingestion.
This mirrors the idle-sleep pattern common in hand-rolled SQS consumers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/impl/aws/s3/input.go, focusing on sqsTargetReader.Pop and the existing 500ms empty-receive throttle. Trace how SQS settings are represented and applied, then verify that a positive sqs.idle_poll_period changes the idle delay while the default 0s preserves current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100