deepset-ai / deepset-ai/haystack-core-integrations

When use Langfuse tracing in large Documents pipeline: task_done() called too many times

Open
#1,641 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

integration:langfuse P3
Dominant language
Python
Stars
203
Forks
332
Avg merge
2d 4h
Merged PRs (30d)
80

Description

Describe the bug
When use Langfuse tracing in large Documents pipline.

2025-04-11 14:49:08 - langfuse - WARNING - Item exceeds size limit (size: 1736099), dropping input / output / metadata of item until it fits.
2025-04-11 14:49:08 - langfuse - WARNING - Item does not have body or input/output fields, dropping item.
Exception in thread Thread-17: 
Traceback (most recent call last):
  File "/root/xxx/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/xxx/lib/python3.10/site-packages/langfuse/_task_manager/ingestion_consumer.py", line 246, in run
2025-04-11 14:49:08 - haystack.core.pipeline.pipeline - INFO - Running component prompt_builder
    self.upload()
  File "/root/xxx/lib/python3.10/site-packages/langfuse/_task_manager/ingestion_consumer.py", line 261, in upload
    self._ingestion_queue.task_done()
  File "/root/xxx/lib/python3.10/queue.py", line 75, in task_done   
    raise ValueError('task_done() called too many times')
ValueError: task_done() called too many times

To Reproduce
code:

        retrieval_pipeline = AsyncPipeline()
        retrieval_pipeline.add_component("tracer", LangfuseConnector("DB"))

Describe your environment (please complete the following information):

langfuse                  2.60.2
langfuse-haystack         0.10.1
haystack-ai               2.12.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LangfuseConnector("DB") entry point used in the AsyncPipeline reproduction and compare it with the ingestion_consumer traceback from langfuse 2.60.2. Reproduce the large Documents pipeline case with the listed Haystack and integration versions, then verify that the queue no longer raises the task_done() exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.