AbsaOSS / AbsaOSS/cobrix

DataBricks Unity Catalog and Cobrix

Open
#665 13 comments 0 reactions 0 assignees View on GitHub
accepted enhancement
Dominant language
Scala
Stars
170
Forks
96
Avg merge
57m
Merged PRs (30d)
2

Description

Hi guys,

thanks a lot for Cobrix. It's really great!

We're moving from Spark (Hadoop) on Premises to DataBricks in the Azure Cloud.
And have encountered a strange problem when using the Unity Catalog.

Both the copybook and the data are stored in a managed Volume in Unity catalog. (Copybooks are simple, no nested fields.) If we do something as simple as
```
df = spark.read.format("cobol"). \
option("copybook", "/Volumes/dev/raw/copybook.cob"). \
load("/Volumes/dev/raw/my.data")
```
in a Python notebook or script, everything works fine if the code runs on a Compute cluster created by the same person who executes the code. If the code is run by Person A on a cluster created by person B, an "Insufficient Permissions" exception is raised.
See
```
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501
File , line 1
----> 1 cobol_import("bnktfili")
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1874, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1871 info = error_details_pb2.ErrorInfo()
1872 d.Unpack(info)
-> 1874 raise convert_exception(
1875 info,
1876 status.message,
1877 self._fetch_enriched_error(info),
1878 self._display_server_stack_trace(),
1879 ) from None
1881 raise SparkConnectGrpcException(status.message) from None
1882 else:
```
Person A has full read permissions on any item in the catalog.
The problem only arrises when using Cobrix. If we just load some CSV or parquet file form a Volume, no such problem occurs.

Any idea what goes on here or what we could do?
Any help is much appreciated. Thanks a lot.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.