populationgenomics / populationgenomics/analysis-runner
Analysis runner bug - assuming config on import
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 5
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 8
Description
Raised by @violetbrina
A couple of the submodules in the analysis runner assume the existence of a config on import which it shouldn't be doing.
Discovered while writing unittests for prod pipes.
Initial offenders found (but there coule be more) are:
- analysis_runner/dataproc.py
_config = get_config()
ACCESS_LEVEL = _config['workflow']['access_level']
DATASET = _config['workflow']['dataset']
DATASET_GCP_PROJECT = _config['workflow']['dataset_gcp_project']
GCLOUD_CONFIG_SET_PROJECT = f'gcloud config set project {DATASET_GCP_PROJECT}'
- analysis_runner/examples/cromwell_from_hail_batch.py
_config = get_config()
BILLING_PROJECT = _config['hail']['billing_project']
DATASET = _config['workflow']['dataset']
ACCESS_LEVEL = _config['workflow']['access_level']
These should be updates to not pull values on import. The examples one is probably less important. But it's worth doing another look to see if any other modules do the same.
I propose a cached getenv() or equivalent function to pull those environment variables rather than try and load on import.
^ That approach seems fine, otherwise would throw in just making it call a function to get those values if required. We'd also need to look for any places that BILLING_PROJECT is imported from analysis_runner.dataproc for example.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting analysis_runner/dataproc.py and analysis_runner/examples/cromwell_from_hail_batch.py, then search the analysis_runner modules for configuration reads at import time. Check the unittest work for prod pipes and trace any imports of BILLING_PROJECT or related constants. Done means modules can be imported without loading config, while required values are still obtained when used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100