microsoft / microsoft/AzureTRE
Deploying the Databricks Workspace Service should automatically set up a Unity Catalog pointing to the workspace storage account datalake container
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 235
- Forks
- 192
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
Is your feature request related to a problem? Please describe.
Right now accessing workspace storage account data from within Databricks is a little convoluted (if there is an easier way please let me know!):
- Get the workspace storage account data lake endpoint and account key
- Put the account key in a Databricks Secret (or just use it directly in notebooks, although this is definitely not best practice).
- Access the storage account directly from dbutils, e.g.
spark.conf.set(
"fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
dbutils.secrets.get(scope="<scope-name", key="storage-account-access-key-name"))
df = spark.read.json("dbfs:/databricks-datasets/iot/iot_devices.json")
Describe the solution you'd like
It would be a lot easier if the Databricks workspace service automatically created a Unity Catalog pointing to the datalake container.
This would require the following steps to be automated:
- Grant the Databricks managed identity Storage Blob Data Contributor on the workspace storage account.
- There is already a Databricks Credential set up to use the managed identity, so a new one does not need to be created.
- Create an External Location in Databricks Unity Catalog configured to access the
datalakecontainer in the workspace storage account (stgwsNNNN). - Create a Catalog in Unity Catalog configured to access the external location.
- All of the above steps can be performed via Terraform included in the Databricks Workspace Service (there is a first party terraform provider for databricks; their databricks CLI actually uses terraform internally to perform most of its functionality).
This would allow Databricks users to immediately start working with files/data in the datalake container.
Describe alternatives you've considered
Manually accessing the storage account via the code up above works, but is cumbersome and requires multiple manual steps/connections.
Additional context
Unity Catalog is the preferred method for Databricks to access cloud storage at this point.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the AzureTRE Databricks Workspace Service deployment and its existing Databricks Credential and Terraform configuration. Implement the requested automation for the workspace storage account's datalake container: grant the managed identity access, then create the Unity Catalog external location and catalog. Done means a deployed workspace can access that container through Unity Catalog without the manual secret and dbutils steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, terraform
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100