opensearch-project / opensearch-project/data-prepper
Improve the logging in the OSI filtering stage
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
It seems during ingestion stage, the OSI will filter based on certain criteria like start_time, and prefix, we like to see those in logging so we can understand what criteria are applied during ingestion process.
Describe the solution you'd like
Output a log line explain the filtering criteria used.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with S3ScanPartitionCreationSupplier.java at the linked lines 108-117 and trace how the OSI filtering criteria are assembled during ingestion. Add a log line that explains the applied start_time and prefix criteria, then verify that ingestion output includes those criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100