[Python] [Rust] Parquet read file fails with batch size 1_000_000 and 41 row groups

Here is the error I got:

Pyarrow:

```

>>> df = pd.read_parquet("test.parquet", engine="pyarrow")
Traceback (most recent call last):
File "", line 1, in
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 296, in read_parquet
return impl.read(path, columns=columns, \*\*kwargs)
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 125, in read
path, columns=columns, \*\*kwargs
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 1281, in read_table
use_pandas_metadata=use_pandas_metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 1137, in read
use_pandas_metadata=use_pandas_metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 605, in read
table = reader.read(\*\*options)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 253, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 1136, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Unexpected end of stream

```

fastparquet:

```
>>> df = pd.read_parquet("test.parquet", engine="fastparquet")
/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/encoding.py:222: NumbaDeprecationWarning: The 'numba.jitclass' decorator has moved to 'numba.experimental.jitclass' to better reflect the experimental nature of the functionality. Please update your imports to accommodate this change and see for the time frame.
Numpy8 = numba.jitclass(spec8)(NumpyIO)
/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/encoding.py:224: NumbaDeprecationWarning: The 'numba.jitclass' decorator has moved to 'numba.experimental.jitclass' to better reflect the experimental nature of the functionality. Please update your imports to accommodate this change and see for the time frame.
Numpy32 = numba.jitclass(spec32)(NumpyIO)
Traceback (most recent call last):
File "", line 1, in
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 296, in read_parquet
return impl.read(path, columns=columns, \*\*kwargs)
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 201, in read
return parquet_file.to_pandas(columns=columns, \*\*kwargs)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/api.py", line 399, in to_pandas
index=index, assign=parts)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/api.py", line 228, in read_row_group
scheme=self.file_scheme)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 354, in read_row_group
cats, selfmade, assign=assign)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 331, in read_row_group_arrays
catdef=out.get(name+'-catdef', None))
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 245, in read_col
skip_nulls, selfmade=selfmade)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 99, in read_data_page
raw_bytes = _read_page(f, header, metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 31, in _read_page
page_header.uncompressed_page_size)
AssertionError: found 120016208 raw bytes (expected None)

```

The corresponding Rust code is:

```

use parquet::{
column::writer::ColumnWriter::BoolColumnWriter,
column::writer::ColumnWriter::Int32ColumnWriter,
[file::]

{ properties::WriterProperties, writer::

{FileWriter, SerializedFileWriter}

,
},
schema::parser::parse_message_type,
};
use std::\{fs, rc::Rc};

fn main() {
let schema = "
message schema

{ REQUIRED INT32 a; REQUIRED BOOLEAN b; }

";

let schema = Rc::new(parse_message_type(schema).unwrap());
let props = Rc::new(
WriterProperties::builder()
.set_statistics_enabled(false)
.set_dictionary_enabled(false)
.build(),
);
let file = fs::File::create("test.parquet").unwrap();
let mut writer = SerializedFileWriter::new(file, schema, props).unwrap();
let batch_size = 1_000_000;
let mut data = vec![];
let mut data_bool = vec![];
for i in 0..batch_size

{ data.push(i); data_bool.push(true); }

let mut j = 0;
loop {
let mut row_group_writer = writer.next_row_group().unwrap();
let mut col_writer = row_group_writer.next_column().unwrap().unwrap();
if let Int32ColumnWriter(ref mut typed_writer) = col_writer

{ typed_writer.write_batch(&data, None, None).unwrap(); }

else

{ panic!(); }

row_group_writer.close_column(col_writer).unwrap();
let mut col_writer = row_group_writer.next_column().unwrap().unwrap();
if let BoolColumnWriter(ref mut typed_writer) = col_writer \{ typed_writer.write_batch(&data_bool, None, None).unwrap(); } else \{ panic!(); }

row_group_writer.close_column(col_writer).unwrap();
writer.close_row_group(row_group_writer).unwrap();

j += 1;
if j \* batch_size > 40_000_000

{ break; }

}
writer.close().unwrap()
}

```

 

**Reporter**: [Novice](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=novice)
#### Related issues:
- [[C++] Require error message when using ParquetException::EofException](https://github.com/apache/arrow/issues/42957) (relates to)
#### Original Issue Attachments:
- [test_2.parquet.tgz](https://issues.apache.org/jira/secure/attachment/13002100/test_2.parquet.tgz)

_{**Note**: *This issue was originally created as [PARQUET-1858](https://issues.apache.org/jira/browse/PARQUET-1858). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

まず、提供された Rust 再現プログラムをバッチサイズ 1,000,000、41 個の行グループで実行し、次に pyarrow と fastparquet の両方を使って pandas 経由で test.parquet を読み取ってみてください。得られたエラーを添付された test_2.parquet.tgz と比較してください。この issue では、期待される動作が定義されておらず、完了のための対象ファイルやテストも指定されていません。

索引モデルが issue の本文から書いたものです。