redpanda-data / redpanda-data/connect

aws_s3: configurable idle back-off for SQS polling (sqs.idle_poll_period)

Open
#4,478 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Problem

The aws_s3 input, when SQS-driven, throttles empty-queue polling with a hardcoded 500ms between ReceiveMessage calls (internal/impl/aws/s3/input.go, sqsTargetReader.Pop). With short polling that is ~2 requests/s/instance on an idle queue; even with wait_time_seconds: 20 (the SQS maximum) the floor is ~1 request / 20.5s. There is no way to back off further.

For deployments that poll SQS from outside the AWS account (e.g. across a NAT gateway), this idle traffic adds avoidable request volume and NAT data-processing at fleet scale.

Proposal

Add an optional sqs.idle_poll_period duration field. When set (> 0) it replaces the 500ms throttle applied after an empty receive; the default 0s preserves current behaviour exactly. It stacks after wait_time_seconds long-polling and trades first-message latency for fewer requests — suited to batch S3 ingestion.

This mirrors the idle-sleep pattern common in hand-rolled SQS consumers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/impl/aws/s3/input.go, focusing on sqsTargetReader.Pop and the existing 500ms empty-receive throttle. Trace how SQS settings are represented and applied, then verify that a positive sqs.idle_poll_period changes the idle delay while the default 0s preserves current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.