AbsaOSS / AbsaOSS/cobrix

DataBricks Unity Catalog and Cobrix

Aperta
#665 13 commenti 0 reazioni 0 assegnatari Vedi su GitHub
accepted enhancement
Lingua principale
Scala
Stelle
170
Fork
96
Merge medio
57m
PR unite (30g)
2

Descrizione

Hi guys,

thanks a lot for Cobrix. It's really great!

We're moving from Spark (Hadoop) on Premises to DataBricks in the Azure Cloud.
And have encountered a strange problem when using the Unity Catalog.

Both the copybook and the data are stored in a managed Volume in Unity catalog. (Copybooks are simple, no nested fields.) If we do something as simple as
```
df = spark.read.format("cobol"). \
option("copybook", "/Volumes/dev/raw/copybook.cob"). \
load("/Volumes/dev/raw/my.data")
```
in a Python notebook or script, everything works fine if the code runs on a Compute cluster created by the same person who executes the code. If the code is run by Person A on a cluster created by person B, an "Insufficient Permissions" exception is raised.
See
```
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501
File , line 1
----> 1 cobol_import("bnktfili")
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1874, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1871 info = error_details_pb2.ErrorInfo()
1872 d.Unpack(info)
-> 1874 raise convert_exception(
1875 info,
1876 status.message,
1877 self._fetch_enriched_error(info),
1878 self._display_server_stack_trace(),
1879 ) from None
1881 raise SparkConnectGrpcException(status.message) from None
1882 else:
```
Person A has full read permissions on any item in the catalog.
The problem only arrises when using Cobrix. If we just load some CSV or parquet file form a Volume, no such problem occurs.

Any idea what goes on here or what we could do?
Any help is much appreciated. Thanks a lot.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.