GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Connection type FileSystem is supported and the limitations are not documented

Open
#872 0 comments 0 reactions 1 assignee Claimed by @helensilva14 View on GitHub
priority: p3 type: docs type: testing
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

Filesystem (files or GCS location) is a connection type with CSV or JSON files. However there are limitations (see below) and they should be clearly documented after some testing.

We currently ingest the file into a pandas dataframe and use the ibis connector to pandas. There are limitations on what functionality we can support:
1. Some column validations are supported (count, sum etc) - need to validate full list.
2. Custom Query Column Validations are likely supported (I have not checked)
3. Schema validations are not "really" supported because pandas dataframe infers a schema, but that is limited in datatypes
4. Row validations are not supported (result in an exception) - because pandas does not support HashBytes on a series of columns
5. Partitioning is not (and will not) be supported - because we can't read flat files in parts to reduce memory usage.
6. We can only support files that fit into the memory (due to point 5 above)

If customers want to use large files - we can document a workaround to load the file into BigQuery and then use that as a source.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.