apache / apache/beam

[Bug]: DataFrame API `to_feather` fails in most cases

Open
#24,091 1 comment 0 reactions 0 assignees View on GitHub
bug dataframe dsl P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Pandas to_feather fails when used with a non-default index:
```
ValueError: feather does not support serializing a non-default index for the index; you can .reset_index() to make the index into column(s)
```

Unfortunately, our DataFrame partitions will only have a default index in rare cases. Typically each partition will have some hash-based index. So to_feather will usually fail with the above error at execution time.

- https://stackoverflow.com/questions/74345723/how-to-save-deferreddataframe-to-feather-with-dataflowrunner
- https://lists.apache.org/thread/wszft7ts7l4xg05lqfbwybn1mbpkkffz

### Issue Priority

Priority: 2

### Issue Component

Component: dsl-dataframe

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.