astronomy-commons / astronomy-commons/lsdb

Ensure `_healpix_29` is the first column in the arrow schema

Open Beginner friendly
#595 1 comment 0 reactions 0 assignees View on GitHub
bug good first issue
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

In [get_arrow_schema](https://github.com/astronomy-commons/lsdb/blob/7a10776340896cd0abbc2c1955e90f769e443afb/src/lsdb/io/schema.py#L7), there's no guarantee that the spatial index column (`_healpix_29`) appears at index 0 in the schema. This means that, for example, if we read a leaf parquet file using pandas `pd.read_parquet(..)` the spatial index column will show up at the end of the table. I don't think this breaks anything necessarily because we always set the spatial index for each loaded Pandas partition, but it would be nice to have for consistency: in `hats-import` we also insert the spatial index in the first column when [reducing](https://github.com/astronomy-commons/hats-import/blob/0b2352088885385cb81ab41774e597471e3e5e5f/src/hats_import/catalog/map_reduce.py#L292) the shards.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lsdb/io/schema.py at get_arrow_schema and inspect how the schema columns are assembled. Compare the expected ordering with the hats-import reduction behavior described in the issue, then verify that reading a leaf parquet file with pandas places _healpix_29 at index 0 when the change is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.