null_count is omitted for large columns in parquet files
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Currently in [ParquetMetadataConverter.java](https://github.com/apache/parquet-java/blob/7be05b4702df78ae0c0c6b44adc6b7b7af2d931f/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java), there is a guard that prevents the writing of statistics such as min/max AND null_count when the stats are larger than the max allowed size under truncation. The rationale for this makes sense for omitting min/max, however null_count can be written on the file despite the size of its content. See the code below:
https://github.com/apache/parquet-java/blob/7be05b4702df78ae0c0c6b44adc6b7b7af2d931f/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L800-L807
The missing `null_count` metadata sometimes causes downstream consumers of the parquet files to error. For example in Snowflake we are seeing the following kind of error:
```
non-nullable column without default has null values according to file statistics
```
### Component(s)
Core
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.