aws-samples / aws-samples/dbt-glue

Option to deactivate Lake Formation

Open
#658 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the feature
Add an option to `dbt-glue` configuration `profile.yml` file to disable the use of Lake Formation, e.g. `use_lakeformation: bool`.

### Describe alternatives you've considered
None in our case, since we work in a large organization where security won't give us Lake Formation rights, we do just plain ol' glue :)

### Additional context
By default, [dbt-glue tries to use Lake Formation](https://github.com/aws-samples/dbt-glue/blob/1.10.15/dbt/adapters/glue/impl.py#L438) for creating schemas:

```python
def create_schema(self, relation: BaseRelation):
session, client = self.get_connection()
lf = boto3.client("lakeformation", region_name=session.credentials.region)
sts = boto3.client("sts")
...
try:
lf.batch_grant_permissions(CatalogId=account, Entries=Entries)
except Exception as e:
...
```

This generates errors in our logs
```sql
Glue adapter: An error occurred (AccessDeniedException) when calling the BatchGrantPermissions
operation: User: arn:aws:sts::***:assumed-role/***/dbt-client is not authorized to perform:
lakeformation:BatchGrantPermissions on resource: arn:aws:lakeformation:eu-west-3:***:catalog:***
with an explicit deny in a service control policy
```

DBT survives this error then falls back to standard glue client to create the schema.

🔴 **Impact**: noise in our logs, forces us to add exclusions to our alerting, with risk of exclusing legitimate errors.

### Who will this benefit?
All `dbt-glue` users who just want to do "pure" glue without Lake Formation.

### Are you interested in contributing this feature?
Yes

Contributor guide

Open the contributing guide

Research direction

Start in dbt/adapters/glue/impl.py at create_schema and review how dbt-glue reads profile.yml configuration. Trace the existing Lake Formation call and the standard Glue fallback. Done means the configuration can disable Lake Formation attempts for users without permissions while schema creation still uses Glue normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.