AbsaOSS / AbsaOSS/cobrix

DataBricks Unity Catalog and Cobrix

Abierto
#665 13 comentarios 0 reacciones 0 asignados Ver en GitHub
accepted enhancement
Lenguaje dominante
Scala
Estrellas
170
Forks
96
Merge medio
57 min
PR fusionados (30 d)
2

Descripción

Hi guys,

thanks a lot for Cobrix. It's really great!

We're moving from Spark (Hadoop) on Premises to DataBricks in the Azure Cloud.
And have encountered a strange problem when using the Unity Catalog.

Both the copybook and the data are stored in a managed Volume in Unity catalog. (Copybooks are simple, no nested fields.) If we do something as simple as
```
df = spark.read.format("cobol"). \
option("copybook", "/Volumes/dev/raw/copybook.cob"). \
load("/Volumes/dev/raw/my.data")
```
in a Python notebook or script, everything works fine if the code runs on a Compute cluster created by the same person who executes the code. If the code is run by Person A on a cluster created by person B, an "Insufficient Permissions" exception is raised.
See
```
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501
File , line 1
----> 1 cobol_import("bnktfili")
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1874, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1871 info = error_details_pb2.ErrorInfo()
1872 d.Unpack(info)
-> 1874 raise convert_exception(
1875 info,
1876 status.message,
1877 self._fetch_enriched_error(info),
1878 self._display_server_stack_trace(),
1879 ) from None
1881 raise SparkConnectGrpcException(status.message) from None
1882 else:
```
Person A has full read permissions on any item in the catalog.
The problem only arrises when using Cobrix. If we just load some CSV or parquet file form a Volume, no such problem occurs.

Any idea what goes on here or what we could do?
Any help is much appreciated. Thanks a lot.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.