[Python] [Rust] Parquet read file fails with batch size 1_000_000 and 41 row groups

Here is the error I got:

Pyarrow:

```

>>> df = pd.read_parquet("test.parquet", engine="pyarrow")
Traceback (most recent call last):
File "", line 1, in
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 296, in read_parquet
return impl.read(path, columns=columns, \*\*kwargs)
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 125, in read
path, columns=columns, \*\*kwargs
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 1281, in read_table
use_pandas_metadata=use_pandas_metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 1137, in read
use_pandas_metadata=use_pandas_metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 605, in read
table = reader.read(\*\*options)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/pyarrow/parquet.py", line 253, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 1136, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Unexpected end of stream

```

fastparquet:

```
>>> df = pd.read_parquet("test.parquet", engine="fastparquet")
/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/encoding.py:222: NumbaDeprecationWarning: The 'numba.jitclass' decorator has moved to 'numba.experimental.jitclass' to better reflect the experimental nature of the functionality. Please update your imports to accommodate this change and see for the time frame.
Numpy8 = numba.jitclass(spec8)(NumpyIO)
/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/encoding.py:224: NumbaDeprecationWarning: The 'numba.jitclass' decorator has moved to 'numba.experimental.jitclass' to better reflect the experimental nature of the functionality. Please update your imports to accommodate this change and see for the time frame.
Numpy32 = numba.jitclass(spec32)(NumpyIO)
Traceback (most recent call last):
File "", line 1, in
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 296, in read_parquet
return impl.read(path, columns=columns, \*\*kwargs)
File "/home/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 201, in read
return parquet_file.to_pandas(columns=columns, \*\*kwargs)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/api.py", line 399, in to_pandas
index=index, assign=parts)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/api.py", line 228, in read_row_group
scheme=self.file_scheme)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 354, in read_row_group
cats, selfmade, assign=assign)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 331, in read_row_group_arrays
catdef=out.get(name+'-catdef', None))
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 245, in read_col
skip_nulls, selfmade=selfmade)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 99, in read_data_page
raw_bytes = _read_page(f, header, metadata)
File "/home/miniconda3/envs/ds/lib/python3.7/site-packages/fastparquet/core.py", line 31, in _read_page
page_header.uncompressed_page_size)
AssertionError: found 120016208 raw bytes (expected None)

```

The corresponding Rust code is:

```

use parquet::{
column::writer::ColumnWriter::BoolColumnWriter,
column::writer::ColumnWriter::Int32ColumnWriter,
[file::]

{ properties::WriterProperties, writer::

{FileWriter, SerializedFileWriter}

,
},
schema::parser::parse_message_type,
};
use std::\{fs, rc::Rc};

fn main() {
let schema = "
message schema

{ REQUIRED INT32 a; REQUIRED BOOLEAN b; }

";

let schema = Rc::new(parse_message_type(schema).unwrap());
let props = Rc::new(
WriterProperties::builder()
.set_statistics_enabled(false)
.set_dictionary_enabled(false)
.build(),
);
let file = fs::File::create("test.parquet").unwrap();
let mut writer = SerializedFileWriter::new(file, schema, props).unwrap();
let batch_size = 1_000_000;
let mut data = vec![];
let mut data_bool = vec![];
for i in 0..batch_size

{ data.push(i); data_bool.push(true); }

let mut j = 0;
loop {
let mut row_group_writer = writer.next_row_group().unwrap();
let mut col_writer = row_group_writer.next_column().unwrap().unwrap();
if let Int32ColumnWriter(ref mut typed_writer) = col_writer

{ typed_writer.write_batch(&data, None, None).unwrap(); }

else

{ panic!(); }

row_group_writer.close_column(col_writer).unwrap();
let mut col_writer = row_group_writer.next_column().unwrap().unwrap();
if let BoolColumnWriter(ref mut typed_writer) = col_writer \{ typed_writer.write_batch(&data_bool, None, None).unwrap(); } else \{ panic!(); }

row_group_writer.close_column(col_writer).unwrap();
writer.close_row_group(row_group_writer).unwrap();

j += 1;
if j \* batch_size > 40_000_000

{ break; }

}
writer.close().unwrap()
}

```

 

**Reporter**: [Novice](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=novice)
#### Related issues:
- [[C++] Require error message when using ParquetException::EofException](https://github.com/apache/arrow/issues/42957) (relates to)
#### Original Issue Attachments:
- [test_2.parquet.tgz](https://issues.apache.org/jira/secure/attachment/13002100/test_2.parquet.tgz)

_{**Note**: *This issue was originally created as [PARQUET-1858](https://issues.apache.org/jira/browse/PARQUET-1858). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Führe zunächst den bereitgestellten Rust-Reproducer mit einer Batch-Größe von 1.000.000 und 41 Zeilengruppen aus und versuche anschließend, test.parquet über pandas sowohl mit pyarrow als auch mit fastparquet zu lesen. Vergleiche die resultierenden Fehler mit der beigefügten Datei test_2.parquet.tgz; das Issue definiert weder das erwartete Verhalten noch nennt es eine Zieldatei oder einen Test für den Abschluss.

Vom Indexierungsmodell aus dem Issue-Text verfasst.