deepset-ai / deepset-ai/haystack

Feature Request: `pipeline.run()` Progress Bar

Open
#9,314 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

P3
Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Is your feature request related to a problem? Please describe.

As a follow-up of this discussion about the SuperComponent abstraction with @sjrl, I think it would be great to have a progress bar displayed when executing a pipeline with pipeline.run().

Currently, there is no visually indication to a user that the pipeline is executed and which step it is currently running. Even when monitoring and tracing are enabled, the setup requires additional setup time as well as technical in-depth knowledge, while log information is presented in a highly technical format that isn't intuitive for quick status assessment.

There have been some similar requests regarding progress bars in the past, but they are related to specific components or aspects in pipelines (e.g. batch processing, see #2580). Some users also want to have the option to disable progress bars globally (#8782, #5098).

Describe the solution you'd like

A progress bar displayed when running pipeline.run() could

  • provide immediate quick, visual and real-time feedback
  • provide a time estimation and an overall status
  • complement monitoring and tracing
  • increase overall usability

From my point-of-view, I think it would be useful to have this information displayed when pipeline.run():

  • percentage completion
  • processing step indicator (step name, step number of total number of steps)
  • runtime estimation
  • batch processing information (#2580)

Describe alternatives you've considered

Could be integrated directly into the pipeline.run() logic, integrated as parameter to be turned on (True) or off (False) or as env var

Additional context

Hugging Face extensively uses progress bars with tqdm across its ecosystem for model downloading, training, inference, and data processing operations, demonstrating their value in workflows.

Example HF progress bar:
https://stackoverflow.com/questions/74404985/huggingface-transformer-trainer-tqdm-progress-bar-not-moving-at-all-in-jupyter-n

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the pipeline.run() entry point and the linked SuperComponent discussion, then review the related issues #2580, #8782, and #5098. Done should provide clear real-time execution status, step progress, runtime estimation, and an explicit way to control whether the progress bar is shown.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.