Open-EO / Open-EO/openeo-python-client
MultiBackendJobManager.run_jobs() doesn't add new jobs to existing job_tracker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 217
- Forks
- 56
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 2
Description
The MultiBackendJobManager.run_jobs() method takes as input a df, which is a DataFrame containing information about all the jobs to run and an output_file, which contains the path to a csv file to track the status of all the jobs.
If the output_file already exists, however, the run_jobs() method will ignore the df input and continue from the existing jobs in the output_file, as seen in the code below:
output_file = Path(output_file)
if output_file.exists() and output_file.is_file():
# Resume from existing CSV
_log.info(f"Resuming `run_jobs` from {output_file.absolute()}")
df = pd.read_csv(output_file)
status_histogram = df.groupby("status").size().to_dict()
_log.info(f"Status histogram: {status_histogram}")
This makes it so that once a MultiBackendJobManager is run a second time, with the same output_file, it's not possible to add new jobs.
Is is possible that when output_file already exists, run_jobs() creates the union of the input df and existing output_file? Or is there a good reason not to?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating MultiBackendJobManager.run_jobs() and inspect the output_file resume branch shown in the issue. Confirm how the input DataFrame and existing CSV are handled, then ensure rerunning with the same output file preserves existing jobs while including new ones and verify the behavior with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100