dbt-labs / dbt-labs/dbt-adapters

[Bug] Can't set columns in LakeFormation data cell filter with dbt-athena

Open
#937 1 comment 0 reactions 0 assignees View on GitHub
pkg:dbt-athena type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [x] dbt-athena
- [x] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark

### Current Behavior

The `dbt-athena` adapter allows for setting [`column_names`](https://github.com/dbt-labs/dbt-adapters/blob/2a94cc75dba1f98fa5caff1f396f5af7ee444598/dbt-athena/src/dbt/adapters/athena/lakeformation.py#L190) as part of a LakeFormation data cell filter. However, when this is set, as [`ColumnWildcard`](https://github.com/dbt-labs/dbt-adapters/blob/2a94cc75dba1f98fa5caff1f396f5af7ee444598/dbt-athena/src/dbt/adapters/athena/lakeformation.py#L203) is set by default, the following error is raised:
```
An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.
```
This means that it's impossible to set a column restriction as part of a LF data cell filter via a dbt model.

### Expected Behavior

I would expect that it would be possible to set columns as part of a data cell filter definition without raising this error.

### Steps To Reproduce

1. Create a standard dbt project that uses the `dbt-athena` adapter
2. Create a model with the following config:
```
{{
config(
materialized='table',
lf_grants={
'data_cell_filters': {
'enabled': True,
'filters': {
'filter_name': {
'row_filter': 'column1<>1',
'column_names': ['column1', 'column2']
}
}
}
}
)
}}

select 1 as column1, 2 as column2, 3 as column3, 4 as column4
union all
select 2 as column1, 3 as column2, 4 as column3, 5 as column4
```
3. Run `dbt build --select 'model_name'`
4. This should return a failure with the error `An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.`

### Relevant log output

```shell
An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.
```

### Environment

```markdown
- Python:3.9
- dbt-athena-community:1.8.2
- dbt-adapters:1.3.3
```

### Additional Context

See [here](https://github.com/dbt-labs/dbt-adapters/compare/main...moj-analytical-services:dbt-adapters:fix-lf-columns?expand=1) for example fix which I've tested and works.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.