aws-samples / aws-samples/serverless-patterns
New pattern submission - s3-opensearch-serverless-osis-sam
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 1.1k
- Avg merge
- 7d 14h
- Merged PRs (30d)
- 6
Description
## GitHub PR for template:
https://github.com/aws-samples/serverless-patterns/pull/3225
## Title
Amazon S3 to Amazon OpenSearch Serverless with OpenSearch Ingestion
## Description
Ingest JSON objects from Amazon S3 into Amazon OpenSearch Serverless using an Amazon OpenSearch Ingestion (OSIS) pipeline.
When a newline-delimited JSON (JSON Lines) object is uploaded to a source S3 bucket, the bucket publishes an object-created event to an Amazon SQS queue. An OpenSearch Ingestion (OSIS) pipeline uses the Amazon S3 source to poll the queue, read each new object, parse every line into a structured document (newline codec + parse_json processor), and index the documents into an Amazon OpenSearch Serverless collection over SigV4. Documents that fail to index are written to a separate dead-letter S3 bucket (kept separate on purpose so DLQ writes do not re-trigger the source bucket's S3 to SQS notification). The pipeline also publishes audit logs to Amazon CloudWatch Logs and scales automatically between the configured minimum and maximum OpenSearch Compute Units.
The pattern deploys two S3 buckets (source and dead-letter), two SQS queues (ingest and its dead-letter queue), an OpenSearch Serverless collection with encryption, network, and data-access policies, an OpenSearch Ingestion pipeline, a CloudWatch log group, and the IAM role the pipeline assumes. It is deployed with AWS SAM and requires no custom application code.
## Language
YAML
## Framework
AWS SAM
## Level
300
## Services from / to
Amazon S3 to Amazon OpenSearch Serverless (via Amazon OpenSearch Ingestion)
## Additional resources
- Using an OpenSearch Ingestion pipeline with Amazon S3: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-s3.html
- Ingesting data into Amazon OpenSearch Serverless collections: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html
- Amazon OpenSearch Ingestion: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html
## Author bio
Sudhansu - Serverless builder contributing to Serverless Land patterns.
Contributor guide
Research direction
Start with the linked GitHub PR #3225 and review the AWS SAM template for its S3, SQS, OpenSearch Serverless, OpenSearch Ingestion, CloudWatch, and IAM resources. Use the referenced AWS documentation to validate the deployment and ingestion flow; done means the pattern is reviewed and the submitted template works as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, yaml
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100