microsoft / microsoft/AzureTRE
Allow researchers to install packages using the Nexus shared service on Azure Databricks service clusters
Open
Nobody has claimed this yet.
story
- Dominant language
- Python
- Stars
- 235
- Forks
- 192
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
Description
As a researcher / data engineer
I want to be able to install packages using the Nexus shared service on my Azure Databricks clusters
So that I can use it in my notebooks
Acceptance criteria
- User is able to install libraries on databricks cluster published in Nexus shared service
Technical references:
- https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/pypi-repositories#PyPIRepositories-Download,searchandinstallpackagesusingpip
- https://learn.microsoft.com/en-us/azure/databricks/kb/clusters/install-private-pypi-repo
- https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/global_init_script
- script we use for VMs: https://github.com/microsoft/AzureTRE/blob/main/templates/workspace_services/guacamole/user_resources/guacamole-azure-linuxvm/terraform/pypi_sources_config.sh
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Azure Databricks private PyPI guidance and the linked Terraform global_init_script resource, then compare the existing pypi_sources_config.sh used for VMs. Done means a library can be installed on an Azure Databricks service cluster from the Nexus shared service and used in a notebook.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python, terraform
- Domain
- cloud, data-engineering, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100