AbsaOSS / AbsaOSS/cobrix

DataBricks Unity Catalog and Cobrix

Đang mở
#665 13 bình luận 0 reaction 0 người được giao Xem trên GitHub
accepted enhancement
Ngôn ngữ chính
Scala
Star
170
Fork
96
Merge trung bình
57 phút
Pull request đã merge (30 ngày)
2

Mô tả

Hi guys,

thanks a lot for Cobrix. It's really great!

We're moving from Spark (Hadoop) on Premises to DataBricks in the Azure Cloud.
And have encountered a strange problem when using the Unity Catalog.

Both the copybook and the data are stored in a managed Volume in Unity catalog. (Copybooks are simple, no nested fields.) If we do something as simple as
```
df = spark.read.format("cobol"). \
option("copybook", "/Volumes/dev/raw/copybook.cob"). \
load("/Volumes/dev/raw/my.data")
```
in a Python notebook or script, everything works fine if the code runs on a Compute cluster created by the same person who executes the code. If the code is run by Person A on a cluster created by person B, an "Insufficient Permissions" exception is raised.
See
```
[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501
File , line 1
----> 1 cobol_import("bnktfili")
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1874, in SparkConnectClient._handle_rpc_error(self, rpc_error)
1871 info = error_details_pb2.ErrorInfo()
1872 d.Unpack(info)
-> 1874 raise convert_exception(
1875 info,
1876 status.message,
1877 self._fetch_enriched_error(info),
1878 self._display_server_stack_trace(),
1879 ) from None
1881 raise SparkConnectGrpcException(status.message) from None
1882 else:
```
Person A has full read permissions on any item in the catalog.
The problem only arrises when using Cobrix. If we just load some CSV or parquet file form a Volume, no such problem occurs.

Any idea what goes on here or what we could do?
Any help is much appreciated. Thanks a lot.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.