apache / apache/arrow

[C++][Python] ArrowIOError: Invalid Parquet file size is 0 bytes on reading from S3

Open
#24,093 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Priority: Major Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I'm not sure if this issue belongs here or to S3FS library.

The error occurs when reading from partitioned parquet from S3, in case when the "root folder" of the parquet was created manually before writing the parquet there. 

I.e. the steps to reproduce:

 
```java

# 1. Create "folder" s3://bucket.name/data.parquet in e.g. cyberduck app

# 2. Write
table = pa.Table.from_pandas(df)
pq.write_table(table, 's3://bucket.name/data.parquet', partition_cols=[], filesystem=s3fs.S3FileSystem())

# 3. Read
pq.read_table('s3://bucket.name/data.parquet', filesystem=s3fs.S3FileSystem())
# ArrowIOError: Invalid Parquet file size is 0 bytes
```
In case when the table was partitioned by a non-empty set of columns, an error reads: "ValueError: Found files in an intermediate directory".

This is likely due to the fact that S3 does not have "folders" per-se, and various software "mimic" creation of empty folder by writing an empty (zero-size) object to S3. So the parquet confuses this object with the actual contents of the parquet file.

At the same time s3fs library correctly identifies the key as a folder: 
```java

s3fs.S3FileSystem().isdir('s3://bucket.name/data.parquet') # Returns True
```
 

**Reporter**: [Vladimir](https://issues.apache.org/jira/browse/ARROW-7867)

**Note**: *This issue was originally created as [ARROW-7867](https://issues.apache.org/jira/browse/ARROW-7867). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the case using pq.write_table and pq.read_table with s3fs.S3FileSystem(), first with an empty root-folder object and then with partition_cols set. Trace how the Python Parquet read path handles filesystem listings and directory markers; done means both unpartitioned and partitioned reads ignore the zero-size folder marker without breaking real Parquet files.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.