opensearch-project / opensearch-project/data-prepper
[BUG] SQS sink `max_message_size` configuration not working
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The max_message_size configuration parameter in SQS sink has no effect. Messages larger than 256KB are rejected regardless of the configured value.
(https://github.com/opensearch-project/data-prepper/blob/8fa41a1025264c0b1b2365f4e7f9064e8ab7aa13/data-prepper-plugins/sqs-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/sqs/SqsSinkBatch.java#L86)
The MAX_BATCH_SIZE_BYTES is hard-coded.
To Reproduce
Steps to reproduce the behavior:
- Configure SQS sink with max_message_size: 1mb in pipeline configuration
- Send event as 300kbs through the pipeline
- Observe that events are still rejected at 256KB limit
- Check logs -
"message":"Event size exceeds max allowed event size".
Expected behavior
When max_message_size is set to 1MB, the SQS sink should accept messages up to 1MB in size, or the configuration should indicate the SQS cannot accept event more than 256kb.
Screenshots
N/A
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
The issue appears to be in SqsSinkService.java where MAX_EVENT_SIZE and MAX_BYTES_IN_BATCH are hard-coded to 256*1024 bytes, ignoring the configuration parameter.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the referenced SqsSinkBatch.java location and inspect SqsSinkService.java, where the issue reports hard-coded size constants. Reproduce the pipeline using max_message_size: 1mb and a 300KB event, then determine whether the configuration should control the limit or be rejected because of SQS constraints. Done means the configured behavior and error handling match the documented expectation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100