aws-samples / aws-samples/dbt-glue
Option to deactivate Lake Formation
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
Add an option to `dbt-glue` configuration `profile.yml` file to disable the use of Lake Formation, e.g. `use_lakeformation: bool`.
### Describe alternatives you've considered
None in our case, since we work in a large organization where security won't give us Lake Formation rights, we do just plain ol' glue :)
### Additional context
By default, [dbt-glue tries to use Lake Formation](https://github.com/aws-samples/dbt-glue/blob/1.10.15/dbt/adapters/glue/impl.py#L438) for creating schemas:
```python
def create_schema(self, relation: BaseRelation):
session, client = self.get_connection()
lf = boto3.client("lakeformation", region_name=session.credentials.region)
sts = boto3.client("sts")
...
try:
lf.batch_grant_permissions(CatalogId=account, Entries=Entries)
except Exception as e:
...
```
This generates errors in our logs
```sql
Glue adapter: An error occurred (AccessDeniedException) when calling the BatchGrantPermissions
operation: User: arn:aws:sts::***:assumed-role/***/dbt-client is not authorized to perform:
lakeformation:BatchGrantPermissions on resource: arn:aws:lakeformation:eu-west-3:***:catalog:***
with an explicit deny in a service control policy
```
DBT survives this error then falls back to standard glue client to create the schema.
🔴 **Impact**: noise in our logs, forces us to add exclusions to our alerting, with risk of exclusing legitimate errors.
### Who will this benefit?
All `dbt-glue` users who just want to do "pure" glue without Lake Formation.
### Are you interested in contributing this feature?
Yes
Contributor guide
Research direction
Start in dbt/adapters/glue/impl.py at create_schema and review how dbt-glue reads profile.yml configuration. Trace the existing Lake Formation call and the standard Glue fallback. Done means the configuration can disable Lake Formation attempts for users without permissions while schema creation still uses Glue normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100