microsoft / microsoft/AzureTRE

Deploying the Databricks Workspace Service should automatically set up a Unity Catalog pointing to the workspace storage account datalake container

Open
#4,488 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Python
Stars
235
Forks
192
Avg merge
1d 23h
Merged PRs (30d)
13

Description

Is your feature request related to a problem? Please describe.
Right now accessing workspace storage account data from within Databricks is a little convoluted (if there is an easier way please let me know!):

  1. Get the workspace storage account data lake endpoint and account key
  2. Put the account key in a Databricks Secret (or just use it directly in notebooks, although this is definitely not best practice).
  3. Access the storage account directly from dbutils, e.g.
spark.conf.set(
  "fs.azure.account.key.<storage-account-name>.dfs.core.windows.net", 
  dbutils.secrets.get(scope="<scope-name", key="storage-account-access-key-name"))

df = spark.read.json("dbfs:/databricks-datasets/iot/iot_devices.json")

Describe the solution you'd like
It would be a lot easier if the Databricks workspace service automatically created a Unity Catalog pointing to the datalake container.

This would require the following steps to be automated:

  1. Grant the Databricks managed identity Storage Blob Data Contributor on the workspace storage account.
  2. There is already a Databricks Credential set up to use the managed identity, so a new one does not need to be created.
  3. Create an External Location in Databricks Unity Catalog configured to access the datalake container in the workspace storage account (stgwsNNNN).
  4. Create a Catalog in Unity Catalog configured to access the external location.
  5. All of the above steps can be performed via Terraform included in the Databricks Workspace Service (there is a first party terraform provider for databricks; their databricks CLI actually uses terraform internally to perform most of its functionality).

This would allow Databricks users to immediately start working with files/data in the datalake container.

Describe alternatives you've considered
Manually accessing the storage account via the code up above works, but is cumbersome and requires multiple manual steps/connections.

Additional context

Unity Catalog is the preferred method for Databricks to access cloud storage at this point.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the AzureTRE Databricks Workspace Service deployment and its existing Databricks Credential and Terraform configuration. Implement the requested automation for the workspace storage account's datalake container: grant the managed identity access, then create the Unity Catalog external location and catalog. Done means a deployed workspace can access that container through Unity Catalog without the manual secret and dbutils steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, terraform
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.