opensearch-project / opensearch-project/data-prepper
[BUG] parquet records are not completely ingested into the open search severless sink
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
I am seeing parquet records are not completely ingested into the open search severless sink sometimes.
To Reproduce
Steps to reproduce the behavior:
- Go to AWS console
- Click on Open Search Ingestion Pipeline
- Check the document metrics and no documents failed to ingest and dlq is empty, however, I still see some parquet records were not completely ingested.
Expected behavior
The parquet record read count need to be the same with document write count. Would also like to see a metrics to reflect how many parquet records are ingested, so I can be confident that all records have been successfully read.
Screenshots
Environment (please complete the following information):
Additional context
Opened a internal ticket as well, will link this issue to the internal ticket.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is named. Start by reproducing the issue through the AWS OpenSearch Ingestion Pipeline and compare parquet record read counts with document write counts and DLQ metrics. Done means the counts match or any discrepancy is accounted for, with a metric showing how many parquet records were ingested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100