astronomy-commons / astronomy-commons/lsdb
Ensure `_healpix_29` is the first column in the arrow schema
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
In [get_arrow_schema](https://github.com/astronomy-commons/lsdb/blob/7a10776340896cd0abbc2c1955e90f769e443afb/src/lsdb/io/schema.py#L7), there's no guarantee that the spatial index column (`_healpix_29`) appears at index 0 in the schema. This means that, for example, if we read a leaf parquet file using pandas `pd.read_parquet(..)` the spatial index column will show up at the end of the table. I don't think this breaks anything necessarily because we always set the spatial index for each loaded Pandas partition, but it would be nice to have for consistency: in `hats-import` we also insert the spatial index in the first column when [reducing](https://github.com/astronomy-commons/hats-import/blob/0b2352088885385cb81ab41774e597471e3e5e5f/src/hats_import/catalog/map_reduce.py#L292) the shards.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lsdb/io/schema.py at get_arrow_schema and inspect how the schema columns are assembled. Compare the expected ordering with the hats-import reduction behavior described in the issue, then verify that reading a leaf parquet file with pandas places _healpix_29 at index 0 when the change is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100