dbt-labs / dbt-labs/dbt-adapters
[Bug] Can't set columns in LakeFormation data cell filter with dbt-athena
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [x] dbt-athena
- [x] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark
### Current Behavior
The `dbt-athena` adapter allows for setting [`column_names`](https://github.com/dbt-labs/dbt-adapters/blob/2a94cc75dba1f98fa5caff1f396f5af7ee444598/dbt-athena/src/dbt/adapters/athena/lakeformation.py#L190) as part of a LakeFormation data cell filter. However, when this is set, as [`ColumnWildcard`](https://github.com/dbt-labs/dbt-adapters/blob/2a94cc75dba1f98fa5caff1f396f5af7ee444598/dbt-athena/src/dbt/adapters/athena/lakeformation.py#L203) is set by default, the following error is raised:
```
An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.
```
This means that it's impossible to set a column restriction as part of a LF data cell filter via a dbt model.
### Expected Behavior
I would expect that it would be possible to set columns as part of a data cell filter definition without raising this error.
### Steps To Reproduce
1. Create a standard dbt project that uses the `dbt-athena` adapter
2. Create a model with the following config:
```
{{
config(
materialized='table',
lf_grants={
'data_cell_filters': {
'enabled': True,
'filters': {
'filter_name': {
'row_filter': 'column1<>1',
'column_names': ['column1', 'column2']
}
}
}
}
)
}}
select 1 as column1, 2 as column2, 3 as column3, 4 as column4
union all
select 2 as column1, 3 as column2, 4 as column3, 5 as column4
```
3. Run `dbt build --select 'model_name'`
4. This should return a failure with the error `An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.`
### Relevant log output
```shell
An error occurred (InvalidInputException) when calling the CreateDataCellsFilter operation: Data cells filter cannot have both column wildcard expression and column names.
```
### Environment
```markdown
- Python:3.9
- dbt-athena-community:1.8.2
- dbt-adapters:1.3.3
```
### Additional Context
See [here](https://github.com/dbt-labs/dbt-adapters/compare/main...moj-analytical-services:dbt-adapters:fix-lf-columns?expand=1) for example fix which I've tested and works.
Contributor guide
Assessment
This issue has not been assessed yet.