awslabs / awslabs/graphrag-toolkit
[FEATURE] Support Amazon OpenSearch Service (managed, non-serverless) with SigV4/IAM auth
- Dominant language
- Python
- Stars
- 442
- Forks
- 106
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 41
Description
### Package
lexical-graph
### Problem statement
The `opensearch://` vector store (added in #397) connects to OpenSearch endpoints that don't use AWS authentication — HTTP basic auth from `OPENSEARCH_USERNAME`/`OPENSEARCH_PASSWORD`, or no auth. It's been validated against a localhost Docker/Finch container. Two deployment shapes aren't covered yet:
- A managed **Amazon OpenSearch Service** domain (not Serverless) that authenticates with IAM/SigV4.
- Any auth beyond basic/none (e.g. mutual TLS).
Raised in review of #397 (@acarbonetto): OpenSearch can run well beyond localhost (EC2, on-prem, managed Service), and we should support more of those once the localhost path is in.
### Proposed solution
Decouple the two concerns the current single flag ties together:
- **Auth mechanism** — `is_sigv4_auth` selects AWS SigV4 vs basic/no auth in `create_os_client` / `create_os_async_client`.
- **AOSS document handling** — the same flag currently also sets the llama-index client's `is_aoss` (custom `id` field, no post-ingest `refresh()`), via `DummyAuth(service='aoss')`.
A managed Amazon OpenSearch Service domain needs SigV4 auth **with** non-AOSS document handling (`_id` + `refresh()`), so these can't stay coupled. Scope:
- Separate the SigV4-auth signal from the AOSS-behavior signal so a managed OpenSearch Service domain can use SigV4 without AOSS document semantics.
- Add a connection path / config for managed Amazon OpenSearch Service (IAM/SigV4), distinct from both `aoss://` and the basic-auth `opensearch://`.
- Decide how the endpoint scheme is expressed (opensearch-py accepts an endpoint without an explicit `http://`/`https://`); document the default.
- Tests for the managed-Service auth path and document handling.
### Alternatives considered
Leave `opensearch://` as basic-auth/no-auth only and document managed Amazon OpenSearch Service as unsupported (current state in #397). Works for self-hosted OpenSearch but excludes managed non-serverless domains.
Contributor guide
Assessment
This issue has not been assessed yet.