Lack of symmetry between shapes and points in on-disk format
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
(Feel free to transfer this issue if it'd be more appropriate in https://github.com/scverse/spatialdata-io.)
While working on https://github.com/HelenaLC/SpatialData/pull/198, I noticed that shapes and points are not treated the same in the on-disk representation of SpatialData:
- shapes are in
shapes/layer_name/shapes.parquet - points are
points/layer_name/points.parquet/part.0.parquet
Is this intended & desirable that the actual parquet file is stored one level deeper for points than for shapes? Conceptually, I don't see any reasons why they shouldn't be stored based exactly on the same path constructor. The current situation seems like a technical constraint has driven the file tree structure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the current on-disk paths for shapes/layer_name/shapes.parquet and points/layer_name/points.parquet/part.0.parquet, including the path constructor used for each. First determine whether the differing Parquet layout is an intentional technical constraint; done means reaching a project decision on whether both representations should use the same path structure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100