opensearch-project / opensearch-project/data-prepper
Support regex patterns for the exclude_suffix in s3 scan
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
As a user of the s3 source with scan, I would like to be able to exclude objects from processing based on a regex pattern instead of a static string. This would allow me to filter out all objects from folders like
exclude_suffix:
- "folder-name/.*"
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the S3 source scan implementation and the existing handling of the exclude_suffix configuration. Trace how suffix values are matched, then review the S3 source tests to identify the expected behavior for regex patterns such as "folder-name/.*". Done means matching objects can be excluded by regex without breaking static-string exclusions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100