Open-EO / Open-EO/openeo-python-client

MultiBackendJobManager.run_jobs() doesn't add new jobs to existing job_tracker

Open
#558 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

collect feedback documentation feature request usability
Dominant language
Python
Stars
217
Forks
56
Avg merge
1d 22h
Merged PRs (30d)
2

Description

The MultiBackendJobManager.run_jobs() method takes as input a df, which is a DataFrame containing information about all the jobs to run and an output_file, which contains the path to a csv file to track the status of all the jobs.
If the output_file already exists, however, the run_jobs() method will ignore the df input and continue from the existing jobs in the output_file, as seen in the code below:

output_file = Path(output_file)
 if output_file.exists() and output_file.is_file():
      # Resume from existing CSV
      _log.info(f"Resuming `run_jobs` from {output_file.absolute()}")
      df = pd.read_csv(output_file)
      status_histogram = df.groupby("status").size().to_dict()
      _log.info(f"Status histogram: {status_histogram}")

This makes it so that once a MultiBackendJobManager is run a second time, with the same output_file, it's not possible to add new jobs.
Is is possible that when output_file already exists, run_jobs() creates the union of the input df and existing output_file? Or is there a good reason not to?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating MultiBackendJobManager.run_jobs() and inspect the output_file resume branch shown in the issue. Confirm how the input DataFrame and existing CSV are handled, then ensure rerunning with the same output file preserves existing jobs while including new ones and verify the behavior with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.