opensearch-project / opensearch-project/data-prepper

[BUG] SQS sink `max_message_size` configuration not working

Open
#6,097 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
The max_message_size configuration parameter in SQS sink has no effect. Messages larger than 256KB are rejected regardless of the configured value.

(https://github.com/opensearch-project/data-prepper/blob/8fa41a1025264c0b1b2365f4e7f9064e8ab7aa13/data-prepper-plugins/sqs-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/sqs/SqsSinkBatch.java#L86)
The MAX_BATCH_SIZE_BYTES is hard-coded.

To Reproduce
Steps to reproduce the behavior:

  1. Configure SQS sink with max_message_size: 1mb in pipeline configuration
  2. Send event as 300kbs through the pipeline
  3. Observe that events are still rejected at 256KB limit
  4. Check logs - "message":"Event size exceeds max allowed event size".

Expected behavior
When max_message_size is set to 1MB, the SQS sink should accept messages up to 1MB in size, or the configuration should indicate the SQS cannot accept event more than 256kb.

Screenshots
N/A

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
The issue appears to be in SqsSinkService.java where MAX_EVENT_SIZE and MAX_BYTES_IN_BATCH are hard-coded to 256*1024 bytes, ignoring the configuration parameter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced SqsSinkBatch.java location and inspect SqsSinkService.java, where the issue reports hard-coded size constants. Reproduce the pipeline using max_message_size: 1mb and a 300KB event, then determine whether the configuration should control the limit or be rejected because of SQS constraints. Done means the configured behavior and error handling match the documented expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.