galaxyproject / galaxyproject/pulsar
Skip postprocessing POST retries when the file does not exist on the pulsar machine
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 15
Description
Setting `max_retries` to retry posts of output files to galaxy is extremely useful, since galaxy is sometimes restarting or too busy to receive the post. The retries also occur when the file does not exist on pulsar and this is not useful, because if the file does not exist upon the completion of a job it will not exist X retries later. Most often the output files are missing because a job has failed. Depending on the settings and the number of expected outputs, a user might have to wait over an hour to find out that their job has failed. Nonexistence of expected output files could be handled by a separate check, prior to the retry loop, and retries skipped in this case.
Contributor guide
Research direction
No file or test is named. Start by tracing the postprocessing path that handles max_retries and posts output files to Galaxy, then identify where file existence can be checked before retrying. Done means missing output files bypass the retry loop while transient Galaxy posting failures still retry; add or update coverage for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100