Parquet File not readable by Google big query (works with Spark)

Hi
I'm trying to write Avro message to parquet on GCS. These parquet should be query by big query engine who support now parquet.

To do this I'm using Secor a kafka log persister tools from pinterest.

First I didn't notice any problem using Spark the same file can be read without any problem all is working perfect.

Now using Big query bring and error like this :
Error while reading table: , error message: Read less values than expected: Actual: 29333, Expected: 33827. Row group: 0, Column: , File:

After investigation using parquet-tools I figured out that in parquet there is metadata regarding number total of unique values for each columns eg from parquet-tools
page 0: DLE:BIT_PACKED RLE:BIT_PACKED [more]... CRC:[PAGE CORRUPT] VC:547

So the VC value indicate that the total number of unique value in the file is 547.

Now when make a spark SQL like SELECT DISTINCT COUNT(column) FROM ... I get 421 mean this number in the metadata is incorrect.

So what is not a problem for Spark to read is a blocking problem for Big data because it relies on these values and found it incorrect.

Is there any configuration of the writer that can prevent these errors in the metadata ? Or is it a normal behavior that should be a problem ?

Thanks

**Environment**: [secor|https://github.com/pinterest/secor]

GCP 

Big Query google cloud

Parquet writer 1.11

 
**Reporter**: [Richard Grossman](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=richiesgr)

_{**Note**: *This issue was originally created as [PARQUET-1946](https://issues.apache.org/jira/browse/PARQUET-1946). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

未指定任何仓库文件或测试。首先使用 Parquet writer 1.11 重现 Secor 的输出，并在 BigQuery 中读取该输出，然后将报告的元数据与 Spark 和 parquet-tools 的结果进行比较。完成的标准是识别出不兼容性，并通过一个针对性的回归用例或有文档记录的限制加以确认。

由索引模型根据 Issue 内容生成。