aws-samples / aws-samples/serverless-patterns

New pattern submission - s3-opensearch-serverless-osis-sam

Open
#3,226 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
1.1k
Avg merge
7d 14h
Merged PRs (30d)
6

Description

## GitHub PR for template:

https://github.com/aws-samples/serverless-patterns/pull/3225

## Title

Amazon S3 to Amazon OpenSearch Serverless with OpenSearch Ingestion

## Description

Ingest JSON objects from Amazon S3 into Amazon OpenSearch Serverless using an Amazon OpenSearch Ingestion (OSIS) pipeline.

When a newline-delimited JSON (JSON Lines) object is uploaded to a source S3 bucket, the bucket publishes an object-created event to an Amazon SQS queue. An OpenSearch Ingestion (OSIS) pipeline uses the Amazon S3 source to poll the queue, read each new object, parse every line into a structured document (newline codec + parse_json processor), and index the documents into an Amazon OpenSearch Serverless collection over SigV4. Documents that fail to index are written to a separate dead-letter S3 bucket (kept separate on purpose so DLQ writes do not re-trigger the source bucket's S3 to SQS notification). The pipeline also publishes audit logs to Amazon CloudWatch Logs and scales automatically between the configured minimum and maximum OpenSearch Compute Units.

The pattern deploys two S3 buckets (source and dead-letter), two SQS queues (ingest and its dead-letter queue), an OpenSearch Serverless collection with encryption, network, and data-access policies, an OpenSearch Ingestion pipeline, a CloudWatch log group, and the IAM role the pipeline assumes. It is deployed with AWS SAM and requires no custom application code.

## Language

YAML

## Framework

AWS SAM

## Level

300

## Services from / to

Amazon S3 to Amazon OpenSearch Serverless (via Amazon OpenSearch Ingestion)

## Additional resources

- Using an OpenSearch Ingestion pipeline with Amazon S3: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-s3.html
- Ingesting data into Amazon OpenSearch Serverless collections: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html
- Amazon OpenSearch Ingestion: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html

## Author bio

Sudhansu - Serverless builder contributing to Serverless Land patterns.

Contributor guide

Open the contributing guide

Research direction

Start with the linked GitHub PR #3225 and review the AWS SAM template for its S3, SQS, OpenSearch Serverless, OpenSearch Ingestion, CloudWatch, and IAM resources. Use the referenced AWS documentation to validate the deployment and ingestion flow; done means the pattern is reviewed and the submitted template works as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, yaml
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.