AbsaOSS / AbsaOSS/cobrix

DataBricks Unity Catalog and Cobrix

Aberta
#665 13 comentários 0 reações 0 responsáveis Ver no GitHub
accepted enhancement
Linguagem predominante
Scala
Estrelas
170
Forks
96
Merge médio
57min
PRs com merge (30d)
2

Descrição

Hi guys,

thanks a lot for Cobrix. It's really great!

We're moving from Spark (Hadoop) on Premises to DataBricks in the Azure Cloud.
And have encountered a strange problem when using the Unity Catalog.

Both the copybook and the data are stored in a managed Volume in Unity catalog. (Copybooks are simple, no nested fields.) If we do something as simple as
```
df = spark.read.format("cobol"). \
option("copybook", "/Volumes/dev/raw/copybook.cob"). \
load("/Volumes/dev/raw/my.data")
```
in a Python notebook or script, everything works fine if the code runs on a Compute cluster created by the same person who executes the code. If the code is run by Person A on a cluster created by person B, an "Insufficient Permissions" exception is raised.
See
```
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501
File , line 1
----> 1 cobol_import("bnktfili")
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1874, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1871 info = error_details_pb2.ErrorInfo()
1872 d.Unpack(info)
-> 1874 raise convert_exception(
1875 info,
1876 status.message,
1877 self._fetch_enriched_error(info),
1878 self._display_server_stack_trace(),
1879 ) from None
1881 raise SparkConnectGrpcException(status.message) from None
1882 else:
```
Person A has full read permissions on any item in the catalog.
The problem only arrises when using Cobrix. If we just load some CSV or parquet file form a Volume, no such problem occurs.

Any idea what goes on here or what we could do?
Any help is much appreciated. Thanks a lot.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.