GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Connection type FileSystem is supported and the limitations are not documented
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
Filesystem (files or GCS location) is a connection type with CSV or JSON files. However there are limitations (see below) and they should be clearly documented after some testing.
We currently ingest the file into a pandas dataframe and use the ibis connector to pandas. There are limitations on what functionality we can support:
1. Some column validations are supported (count, sum etc) - need to validate full list.
2. Custom Query Column Validations are likely supported (I have not checked)
3. Schema validations are not "really" supported because pandas dataframe infers a schema, but that is limited in datatypes
4. Row validations are not supported (result in an exception) - because pandas does not support HashBytes on a series of columns
5. Partitioning is not (and will not) be supported - because we can't read flat files in parts to reduce memory usage.
6. We can only support files that fit into the memory (due to point 5 above)
If customers want to use large files - we can document a workaround to load the file into BigQuery and then use that as a source.
Contributor guide
Assessment
This issue has not been assessed yet.