aws-samples / aws-samples/dbt-glue

Using Lake Formation for permissions causes errors

Open
#156 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
In the situation where:
- Permissions are granted through LakeFormation
- Direct access to objects in S3 is prevented through bucket policy

We notice that data is accessible by using Glue Dynamic Frames, but dbt-glue fails to load it.

Analysis:
The following works:
```
dyf = glueContext.create_dynamic_frame.from_catalog(database='db_name', table_name='table_name')
dyf.printSchema()
df = dyf.toDF()
df.show()
# will show data
```

while the following does not (which is what dbt-glue uses):
```
df1 = spark.sql("select * from db_name.table_name")
df1.show()
```
which will raise the following error:
```
Output exceeds the size limit. Open the full output data in a text editor
Py4JJavaError: An error occurred while calling o89.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 6) (172.35.226.176 executor 1): org.apache.spark.sql.execution.datasources.FileDownloadException: Failed to download file path: s3://path/to/data/part-00000-79859737-058a-46e7-b188-87fe1da90d08-c000.snappy.parquet, range: 0-2429, partition values: [empty row], isDataPresent: false, eTag: fefda370647289be9a7f227db299e6b2-1
at org.apache.spark.sql.execution.datasources.AsyncFileDownloader.next(AsyncFileDownloader.scala:142)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.getNextFile(FileScanRDD.scala:279)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:201)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:159)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:606)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:350)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
...
at org.apache.spark.sql.execution.datasources.AsyncFileDownloader$$anon$1.call(AsyncFileDownloader.scala:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
```

The following (going through a dynamic frame, creating a temp view _and only then_ using `spark.sql`) , however, will work:
```
dyf = glueContext.create_dynamic_frame.from_catalog(database='db_name', table_name='table_name')
df = dyf.toDF()
df.createOrReplaceTempView("my_table")
df0 = spark.sql("select * from my_table")
df0.show()
```

### Steps To Reproduce

- set permissions through LF
- deny direct access to data in bucket through bucket policy
- try to access tables with dbt-glue

=> currently fails

### Expected behavior

The same is expected to work.

Thanks in advance for your help!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by reproducing the Lake Formation and S3 bucket-policy setup, then compare the dbt-glue path using spark.sql with the working Dynamic Frame and temporary-view path. Done means dbt-glue can query the table while direct S3 access remains denied.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
authorization, cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.