aws-samples / aws-samples/bedrock-chat

Sync shows SUCCEEDED when deleting/adding knowledge files, but changes are not reflected due to a race condition

Open
#1,109 0 comments 0 reactions 0 assignees View on GitHub
needs-triage
Dominant language
TypeScript
Stars
1.3k
Forks
535
Avg merge
1d 12h
Merged PRs (30d)
10

Description

## 🐞 Describe the bug

After deleting a knowledge file from a bot, the UI shows sync as completed ("SUCCEEDED"),
but the target document remains indexed in Bedrock Knowledge Base and continues to be
referenced in chat responses.

The same race condition can also occur when **adding** documents, not only when deleting.

**Root Cause**

Two issues combine to produce this behavior:

**1. `handle_check` treats `"INDEXED"` as an unexpected error during deletion**
(`backend/embedding_statemachine/bedrock_knowledge_base/synchronize_data_source.py`)

After `delete_knowledge_base_documents` is called, Bedrock processes the deletion
asynchronously. If `handle_check` polls before Bedrock transitions the document status
from `"INDEXED"` to `"DELETING"`, the status `"INDEXED"` hits the `case _` branch and
raises a generic `Exception` instead of `RetryException`.

```python
# Current code – "INDEXED" falls through to the error case
match status:
case "NOT_FOUND":
pass
case "PENDING" | "DELETING" | "DELETE_IN_PROGRESS":
raise RetryException()
case _:
raise Exception(f"File '{uri}': Bad status '{status}'.") # ← bug
````

**2. `addCatch(ingestionComplete)` silently converts all non-retry exceptions to SUCCEEDED** (`cdk/lib/constructs/embedding.ts`)

The Step Functions state machine only retries on `RetryException`. Any other exception is caught by `addCatch(ingestionComplete)`, where `ingestionComplete` is a `sfn.Pass` state, causing the execution to finish with status `SUCCEEDED` even though the document was never actually deleted.

```typescript
checkIngestionJob
.addRetry({ errors: ['RetryException'], ... })
.addCatch(ingestionComplete, { ... }) // ← catches all other exceptions → Pass → SUCCEEDED
```

## 🔄 To Reproduce

1. Open a bot that has at least one knowledge file indexed in Bedrock Knowledge Base.
2. Navigate to the bot's knowledge management screen and delete one of the files.
3. Wait for the sync status to show "Completed" on the UI.
4. Ask the bot a question that would be answered using the deleted file.
5. Observe that the bot still returns answers based on the deleted document (the file remains indexed in the Knowledge Base).

> **Note:** This issue is timing-dependent and does not occur on every attempt. It is more likely to occur when Bedrock is under load or when the deletion status update is delayed.

## 📷 Screenshots

N/A

## 🔎 Logs for Bot Creation/Update Issues

N/A

## 📝 Additional context

**Proposed fix**

**Fix 1 (primary) – `synchronize_data_source.py`** Add `"INDEXED"` to the retryable status set for deleted documents. When `"INDEXED"` is returned after a delete request, it indicates Bedrock has not yet reflected the deletion — retrying is the correct behavior.

```python
case "NOT_FOUND":
pass

# Add "INDEXED" – means deletion accepted but status not yet updated (eventual consistency)
case "PENDING" | "DELETING" | "DELETE_IN_PROGRESS" | "INDEXED":
raise RetryException()

case _:
raise Exception(f"File '{uri}': Bad status '{status}'.")
```

**Fix 2 (failsafe) – `embedding.ts`** Remove `.addCatch(ingestionComplete)` from `checkIngestionJob`. Unexpected exceptions will then propagate to the outer `mapIngestionJobsForCustomBot.addCatch(syncCustomBotFallback)` (or the shared KB equivalent), which correctly updates the bot status to `FAILED`.

```typescript
// Remove the addCatch so unexpected errors are not silently swallowed
checkIngestionJob.addRetry({
errors: ['RetryException'],
interval: Duration.seconds(15),
maxAttempts: timeout.toSeconds() / 15,
backoffRate: 1,
})
// No addCatch here – unexpected exceptions propagate to the Map state's addCatch(fallback)
```

Fix 1 resolves the root race condition; Fix 2 provides defense-in-depth so that any remaining unexpected exception causes a visible `FAILED` status instead of a silent `SUCCEEDED`.

Contributor guide

Open the contributing guide

Research direction

Start with backend/embedding_statemachine/bedrock_knowledge_base/synchronize_data_source.py and cdk/lib/constructs/embedding.ts, then trace the deletion polling and Step Functions error paths. Verify that an INDEXED status during deletion is retried and that unexpected errors reach the fallback failure state rather than SUCCEEDED. Reproduce or test deletion and addition flows to confirm the knowledge base and UI status agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
ai, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.