GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
schema validation: Nullable=False not identified on Hive
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
Hive uses DESCRIBE to get column information. NOT NULL constraints are displayed after column information of DESCRIBE FORMATTED output:
```
describe formatted pso_data_validator.dvt_null_not_null;
...
| # Constraints | NULL | NULL |
| | NULL | NULL |
| # Not Null Constraints | NULL | NULL |
| Table: | pso_data_validator.dvt_null_not_null | NULL |
| Constraint Name: | nn_100000611_1691424251942_0 | NULL |
| Column Name: | col_src_nn_trg_n | NULL |
| | NULL | NULL |
| Constraint Name: | nn_130700431_1691424251935_0 | NULL |
| Column Name: | col_nn | NULL |
```
This information is not parsed by the version of Ibis we currently use and, as far as I can tell, it is not by the latest version. But at least the latest version has code for parsing output of DESCRIBE FORMATTED (`MetadataParser`) so that would be a better version to tackle this problem on.
If we do decide to proceed with this then look at implementing `test_schema_validation_not_null_vs_nullable()` in `test_hive.py`.
Contributor guide
Assessment
This issue has not been assessed yet.