opensearch-project / opensearch-project/data-prepper
[BUG] DocumentDB source streaming failed to start when collection is empty
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
When both export and stream is enabled, and DocumentDB database or collection is invalid or collection doesn't contain any records, the DocumentDB source fails start stream processing or fails to recover when database or collection is created. Since the collection has no records, all new updates should be synced through stream. Currently the export partition worker job is stuck waiting for partitions. The stream job is stuck waiting for export to be complete. This is a corner case we should handle
Expected behavior
The stream job should not wait if there are no records to export and stream job should process when new change events are available.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the DocumentDB source's export partition worker and stream job behavior when the database or collection is invalid or empty. Reproduce the case with no records, then verify that the stream job does not wait for export and processes newly available change events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100