Add Auth to secure communication with Unity Catalog API

Is your feature request related to a problem? Please describe.

Currently the OSS unity catalog API exposes its server to any requests without requiring authentication or authorization, leaving it very vulnerable outside of basic network security measures like a firewall.

Describe the solution you would like

For a centralized API that is critical to governance, risk, and compliance, the Unity Catalog API should implement both authentication and authorization. The solution should be common and as standard as possible to reduce friction for organizations to integrate with other software.

I recommend using a well understood, open, token format like Bearer or JSON Web Tokens (JWT) (RFC 7519): https://jwt.io/

There are many OSS libraries available for Java/Scala to handle JWTs and OAuth flows, such as auth0/java-jwt.

Auth configuration could be added to the server.properties, including settings such as:

• RSA Private Key (needed for signing tokens)
• RSA Public Key (needed for verifying tokens)
• Time-to-Live (TTL)
• Authn credentials (could be hardcoded in the config, or persisted from some other means)

Note: above settings assume RSA is used for the signing algorithm; depending on how flexible this feature needs to be, there could be optional configuration to choose other algorithms for JWT as well, such as HMAC or ECDSA, as well as configuring the size such as RS256 vs. RSA512, etc.

We will also need to understand how engines like Apache Spark would integrate this security feature. A typical OAuth flow for this would at minimum need to include a new API endpoint for obtaining an access token, and optionally a API endpoint for refreshing tokens as well. To illustrate this flow and how integrations might look:

Example server.properties:

auth.rsaPrivateKey=/etc/unitycatalog/keys/unitycatalog
auth.rsaPublicKey=/etc/unitycatalog/keys/unitycatalog.pub
auth.ttlMinutes=60

auth.clients.0.clientId=000
auth.clients.0.secret=f9872bb90a88bab8b1b19a4de23bae900882cb1b546fe0115d38c68f2ac3b083 # HASH

auth.clients.1.clientId=001
auth.clients.1.secret=b6541aabb2b5dc73042a94ee6f851af648c7eac844f58e4d3a15952f908f9673 # HASH

Example API endpoint for generating access tokens:

curl -X POST http://localhost:8080/api/2.1/unity-catalog/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials" \
  -d "client_id=001" \
  -d "client_secret=keepitsecretkeepitsafe"

Example Spark Configuration:

...
conf.set('spark.sql.catalog.unity', 'org.apache.spark.sql.delta.catalog.DeltaCatalog')
conf.set('spark.sql.catalog.unity.uri', 'http://localhost:8080/api/2.1/unity-catalog/')
conf.set('spark.sql.catalog.unity.auth.client', '001')
conf.set('spark.sql.catalog.unity.auth.secret', 'keepitsecretkeepitsafe')
conf.set('spark.sql.catalog.unity.type', 'rest')
...

Additional context

As a separate request, I could see a similar feature being added to optionally configure TLS for the Unity Catalog server; however, for now I think it's much more important to implement application-level security and TLS is most often terminated at ingresses based on how users deploy these types of backend APIs (e.g. terminating TLS with an Application Load Balancer or a reverse proxy like Nginx). TLS at the actual server application level would just be a nice-to-have for true end-to-end encryption.