astronomy-commons / astronomy-commons/lsdb
returned data types are hard for me to deal with
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
When working on a cross match and join with panstarrs catalogs, the returned dataframe has data types that were hard for me to deal with. They play nicely with pandas, so it was hard for me to notice things like double[pyarrow] are not the same as numpy data types. Specifically, I didn't have trouble with them until I put them into my plotting routines (matplotlib) and then I got strange errors which took me a while to figure out were related to the data types.
Not sure what to do with this information, but thought it might be helpful for you to have the feedback. My solution is the following where I take just the columns I want from the lsdb returned: matched_df and converted them to np data types. I am sure there are other ways of handling this that might be more elegant, but this is one functional way.
df_lc = pd.DataFrame({
'flux': pd.to_numeric(matched_df['psfFlux'] * 1e3, errors='coerce').astype(np.float64),
'err': pd.to_numeric(matched_df['psfFluxErr'] * 1e3, errors='coerce').astype(np.float64),
'time': pd.to_numeric(matched_df['obsTime'], errors='coerce').astype(np.float64),
'objectid': matched_df['objectid'].astype(np.int64),
'band': filtername,
'label': matched_df['label'].astype(str)
}).set_index(["objectid", "label", "band", "time"])
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no repository files or tests. Start by reproducing a cross-match or join with Pan-STARRS catalogs and inspect the returned dataframe dtypes before passing data to matplotlib; completion would require an agreed way to handle or expose these types, which the issue does not define.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matplotlib, numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100