[Bug]: Some written Parquet shard files have 0 num_rows
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
After the 2.44.0 release, we have found an issue when writing to Parquet using shards: some files have 0 number of rows.
Steps:
```python
with beam.Pipeline() as p:
records = p | 'Read' >> beam.Create(
[{'name': 'foo', 'age': 10}, {'name': 'bar', 'age': 20}]
)
_ = records | 'Write' >> beam.io.WriteToParquet("filename",
pyarrow.schema(
[('name', pyarrow.binary()), ('age', pyarrow.int64())]
), num_shards=2
)
for filename in ["filename-00000-of-00002", "filename-00001-of-00002"]:
parquet_file = pyarrow.parquet.ParquetFile(filename)
print(filename)
print(parquet_file.metadata)
print()
```
We get one of the files has 0 number of rows:
```
filename-00000-of-00002
created_by: parquet-cpp-arrow version 9.0.0
num_columns: 2
num_rows: 2
num_row_groups: 1
format_version: 2.6
serialized_size: 514
filename-00001-of-00002
created_by: parquet-cpp-arrow version 9.0.0
num_columns: 2
num_rows: 0
num_row_groups: 0
format_version: 2.6
serialized_size: 340
```
Before (in 2.43.0 version), none of the files had 0 number of rows:
```
filename-00000-of-00002
created_by: parquet-cpp-arrow version 9.0.0
num_columns: 2
num_rows: 1
num_row_groups: 1
format_version: 2.6
serialized_size: 512
filename-00001-of-00002
created_by: parquet-cpp-arrow version 9.0.0
num_columns: 2
num_rows: 1
num_row_groups: 1
format_version: 2.6
serialized_size: 512
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.