[Python] Provide a way to restore a schema from its string representation
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
### Motivation
I need to store a schema in a permanent way, as I'm building a process where several CSV files with the same structure will be converted to Parquet over time, but inferring the schema of the data from each file does not guarantee consistency. I could use something like pickle to dump a binary version of the schema but this is not very portable, and a human-readable representation would allow to do simple modifications by hand if the need arises (e.g. adding or removing columns). For this reason I'm trying to save the schema in a json file and then to read it and re-build a pyarrow Schema.
The first step is easy thanks to each type having a nice string representation, but going backwards is harder as I could not find any way of building a type or a Field from a string that did not break on more complex types such as `timestep[ms, tz=utc]`. I ended up having to implement my own function to parse string representations of types and build the appropriate pyarrow objects.
### Idea
The basic idea would be implementing a function that takes a string representation of a type and returns the corresponding pyarrow type object.
Other things that could make this process easier would be a `to_dict()` function for Schema that basically builds the dictionary obtained combining the `names` and `types` lists of a Schema, or a `from_string(s)` function that can reverse the `to_string()` function.
I can contribute my code to convert strings to type objects, but as I've never contributed to this project before I would like some advice on whether this is something desired and what would be the best way to integrate this functionality with the rest of the code base.
### Component(s)
Python
Contributor guide
Research direction
The issue names no files or tests; start by locating the Python component's existing type and Schema string or serialization APIs. Compare the proposed type parser, Schema to_dict(), and from_string() options, then define round-trip tests showing that supported schemas can be restored and manually edited representations are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100