aws-samples / aws-samples/dbt-glue
Using Lake Formation for permissions causes errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
In the situation where:
- Permissions are granted through LakeFormation
- Direct access to objects in S3 is prevented through bucket policy
We notice that data is accessible by using Glue Dynamic Frames, but dbt-glue fails to load it.
Analysis:
The following works:
```
dyf = glueContext.create_dynamic_frame.from_catalog(database='db_name', table_name='table_name')
dyf.printSchema()
df = dyf.toDF()
df.show()
# will show data
```
while the following does not (which is what dbt-glue uses):
```
df1 = spark.sql("select * from db_name.table_name")
df1.show()
```
which will raise the following error:
```
Output exceeds the size limit. Open the full output data in a text editor
Py4JJavaError: An error occurred while calling o89.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 6) (172.35.226.176 executor 1): org.apache.spark.sql.execution.datasources.FileDownloadException: Failed to download file path: s3://path/to/data/part-00000-79859737-058a-46e7-b188-87fe1da90d08-c000.snappy.parquet, range: 0-2429, partition values: [empty row], isDataPresent: false, eTag: fefda370647289be9a7f227db299e6b2-1
at org.apache.spark.sql.execution.datasources.AsyncFileDownloader.next(AsyncFileDownloader.scala:142)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.getNextFile(FileScanRDD.scala:279)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:201)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:159)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:606)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:350)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
...
at org.apache.spark.sql.execution.datasources.AsyncFileDownloader$$anon$1.call(AsyncFileDownloader.scala:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
```
The following (going through a dynamic frame, creating a temp view _and only then_ using `spark.sql`) , however, will work:
```
dyf = glueContext.create_dynamic_frame.from_catalog(database='db_name', table_name='table_name')
df = dyf.toDF()
df.createOrReplaceTempView("my_table")
df0 = spark.sql("select * from my_table")
df0.show()
```
### Steps To Reproduce
- set permissions through LF
- deny direct access to data in bucket through bucket policy
- try to access tables with dbt-glue
=> currently fails
### Expected behavior
The same is expected to work.
Thanks in advance for your help!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Start by reproducing the Lake Formation and S3 bucket-policy setup, then compare the dbt-glue path using spark.sql with the working Dynamic Frame and temporary-view path. Done means dbt-glue can query the table while direct S3 access remains denied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- authorization, cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100