apache / apache/airflow

Support Managed Knowledge Bases in BedrockCreateKnowledgeBaseOperator

Open
#72,592 1 comment 0 reactions 0 assignees View on GitHub
kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Amazon Bedrock Managed Knowledge Bases reached general availability on 2026-06-17. It is a fully managed RAG offering: Bedrock owns the vector store, the ingestion pipeline and the retrieval infrastructure, so there is no self-managed vector store to configure.

It is created through the same `bedrock-agent` `CreateKnowledgeBase` operation as the existing self-managed base. The only differences are the shape of `KnowledgeBaseConfiguration` and the absence of `storageConfiguration`:

```python
bedrock_agent.create_knowledge_base(
name='my-managed-kb',
roleArn=role_arn,
knowledgeBaseConfiguration={
'type': 'MANAGED',
'managedKnowledgeBaseConfiguration': {'embeddingModeType': 'MANAGED'},
}
}
```

`BedrockCreateKnowledgeBaseOperato` currently cannot produce this call:

1. `knowledgeBaseConfiguration` is hardcorded in `execute()`:
````python
knowledge_base_configuration={
'type': 'VECTOR',
'vectorKnowledgeBaseConfiguration': {'embeddingModelArn': self.embedding_model_arn},
}
````
2. `storageConfiguration=self.storage_config` is always passed to the API call.
3. `embedding_model_arn` and `storage_config` are required `__init__` arguments, and neither applies to a managed knowledge base.
Overrading via `create_knowledge_nase_kwards` does not work either. Both `knoeledgeBaseConfiguration` and `storageConfiguration` are already supplied as explicit keyword arguments, so sypplying them again through the kwargs dict raises `TypeError: got multiple values for keyword argument`.

Separaetly, the `wait_for_indexing` / "no such index" retry loop exists purely to work around vector index propagation in a self-managed vector store. It has no meaning for a managed knowledge base, since there is no index for the user to create.

Net effect: there is currently no way to create a managed knowledge base from the amazon provider.

References:
- https://aws.amazon.com/about-aws/whats-new/2026/06/amazon-bedrock-managed-knowledge-base/
- https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_CreateKnowledgeBase.html
- https://aws.amazon.com/blogs/machine-learning/build-enterprise-search-for-agents-with-amazon-bedrock-managed-knowledge-base/

### Use case/motivation

Refreshing a RAF corpus is a scheduled data pipeline concern: the source documents are usually produced or updated by upstream Airflow tasks, so provisioning the knowledge base and driving ingestion belongs in the same DAG. The provider already covers the self-managed path end to end(`BedrockCreateKnowledgeBaseOperator` -> `BedrockCreateDataSourceOperator` -> `BedrockIngestDataOperator` -> `BedrockKnowledgeBaseActiveSensor`). Managed knowledge bases are the missing half of an otherwise complete story, and they are the lower-friction option for teams that do not want to own vector srore.

### Related issues

Not found.

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with BedrockCreateKnowledgeBaseOperator's __init__ and execute() methods, then compare their arguments and call behavior with the linked Bedrock CreateKnowledgeBase API reference. Done means the operator can create both managed and self-managed knowledge bases, while managed bases avoid self-managed storage settings and the vector-index retry behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.