apache / apache/arrow

[Python] Provide a way to restore a schema from its string representation

Open
#34,898 7 comments 2 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

### Motivation
I need to store a schema in a permanent way, as I'm building a process where several CSV files with the same structure will be converted to Parquet over time, but inferring the schema of the data from each file does not guarantee consistency. I could use something like pickle to dump a binary version of the schema but this is not very portable, and a human-readable representation would allow to do simple modifications by hand if the need arises (e.g. adding or removing columns). For this reason I'm trying to save the schema in a json file and then to read it and re-build a pyarrow Schema.

The first step is easy thanks to each type having a nice string representation, but going backwards is harder as I could not find any way of building a type or a Field from a string that did not break on more complex types such as `timestep[ms, tz=utc]`. I ended up having to implement my own function to parse string representations of types and build the appropriate pyarrow objects.

### Idea
The basic idea would be implementing a function that takes a string representation of a type and returns the corresponding pyarrow type object.
Other things that could make this process easier would be a `to_dict()` function for Schema that basically builds the dictionary obtained combining the `names` and `types` lists of a Schema, or a `from_string(s)` function that can reverse the `to_string()` function.

I can contribute my code to convert strings to type objects, but as I've never contributed to this project before I would like some advice on whether this is something desired and what would be the best way to integrate this functionality with the rest of the code base.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by locating the Python component's existing type and Schema string or serialization APIs. Compare the proposed type parser, Schema to_dict(), and from_string() options, then define round-trip tests showing that supported schemas can be restored and manually edited representations are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.