Handle nulls using SnowflakeIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ; the returned data isn't able to list the nulls from snowflake side.
The nulls from snowflake becomes b'N' ; which is a wrong value and gives incorrect value for the column. In case of strings the actual value can also be 'N'. So it becomes tough to handle it in csv_mapper.
To mitigate this we are using query instead of table and using ifnull / coleasce within the select query for String columns. For non-string columns we are converting back b'N' to null or empty as per requirement.
[1] : [https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake](https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake)
Imported from Jira [BEAM-14431](https://issues.apache.org/jira/browse/BEAM-14431). Original Jira may contain additional context.
Reported by: Varun Rauthan.
Contributor guide
Assessment
This issue has not been assessed yet.