[Bug]: DataFrame API `to_feather` fails in most cases
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Pandas to_feather fails when used with a non-default index:
```
ValueError: feather does not support serializing a non-default index for the index; you can .reset_index() to make the index into column(s)
```
Unfortunately, our DataFrame partitions will only have a default index in rare cases. Typically each partition will have some hash-based index. So to_feather will usually fail with the above error at execution time.
- https://stackoverflow.com/questions/74345723/how-to-save-deferreddataframe-to-feather-with-dataflowrunner
- https://lists.apache.org/thread/wszft7ts7l4xg05lqfbwybn1mbpkkffz
### Issue Priority
Priority: 2
### Issue Component
Component: dsl-dataframe
Contributor guide
Assessment
This issue has not been assessed yet.