opensearch-project / opensearch-project/data-prepper
[BUG] S3 source on_error: delete_messages retains for S3 failures
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The s3 source has a configuration on_error. This can be set to delete_messages which signals an intent to delete messages which failed to parse.
Presently, this feature only works when failing to parse the SQS message itself. If the S3 object fails to parse, then the message is retained.
Expected behavior
I would expect that any error (SQS or S3) results in deleting the message when or_error: delete_messages is set.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the S3 source handling for on_error and compare SQS-message parse failures with S3-object parse failures. Add a regression test covering delete_messages for an S3 failure and verify that the message is deleted while other on_error behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100