deepset-ai / deepset-ai/haystack-core-integrations
When use Langfuse tracing in large Documents pipeline: task_done() called too many times
Open
Nobody has claimed this yet.
integration:langfuse
P3
- Dominant language
- Python
- Stars
- 203
- Forks
- 332
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 80
Description
Describe the bug
When use Langfuse tracing in large Documents pipline.
2025-04-11 14:49:08 - langfuse - WARNING - Item exceeds size limit (size: 1736099), dropping input / output / metadata of item until it fits.
2025-04-11 14:49:08 - langfuse - WARNING - Item does not have body or input/output fields, dropping item.
Exception in thread Thread-17:
Traceback (most recent call last):
File "/root/xxx/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/xxx/lib/python3.10/site-packages/langfuse/_task_manager/ingestion_consumer.py", line 246, in run
2025-04-11 14:49:08 - haystack.core.pipeline.pipeline - INFO - Running component prompt_builder
self.upload()
File "/root/xxx/lib/python3.10/site-packages/langfuse/_task_manager/ingestion_consumer.py", line 261, in upload
self._ingestion_queue.task_done()
File "/root/xxx/lib/python3.10/queue.py", line 75, in task_done
raise ValueError('task_done() called too many times')
ValueError: task_done() called too many times
To Reproduce
code:
retrieval_pipeline = AsyncPipeline()
retrieval_pipeline.add_component("tracer", LangfuseConnector("DB"))
Describe your environment (please complete the following information):
langfuse 2.60.2
langfuse-haystack 0.10.1
haystack-ai 2.12.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LangfuseConnector("DB") entry point used in the AsyncPipeline reproduction and compare it with the ingestion_consumer traceback from langfuse 2.60.2. Reproduce the large Documents pipeline case with the listed Haystack and integration versions, then verify that the queue no longer raises the task_done() exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100