opensearch-project / opensearch-project/data-prepper
[BUG] S3 Scan doesn't scan meta data when no filter is provided
Open
@kkondaka is already working on this.
Since Dec 2, 2025.
bug
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
This config in the S3 Scan works fine
- bucket:
name: offlinebatch
data_selection: metadata_only
filter:
include_prefix:
- bedrockbatch/batch-input
exclude_suffix:
- .out
But without filter, the below doesn't work since no events (metadata only) are sent to the pipeline.
- bucket:
name: offlinebatch
data_selection: metadata_only
To Reproduce
Steps to reproduce the behavior:
1.
Use the below config in the S3 Scan and an ml inference processor
source:
s3:
acknowledgments: true
delete_s3_objects_on_read: false
scan:
buckets:
- bucket:
name: offlinebatch
data_selection: metadata_only
processor:
- ml_inference:
host: "<your host url>"
aws_sigv4: true
action_type: "batch_predict"
service_name: "bedrock"
model_id: "<your model id>"
output_path: "s3://offlinebatch/bedrockbatch/output/"
retry_time_window:
aws:
region: "us-east-1"
ml_when: /bucket == "offlinebatch"
- Once the pipeline started, the ml processor is not triggered and no logs showing the processor has received any events.
- The data_selection: metadata_only might missed creating the metadata events in the S3 Scan if no "filter" is provided.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.