astronomy-commons / astronomy-commons/lsdb

returned data types are hard for me to deal with

Open
#441 9 comments 1 reaction 0 assignees View on GitHub
interface question
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

When working on a cross match and join with panstarrs catalogs, the returned dataframe has data types that were hard for me to deal with. They play nicely with pandas, so it was hard for me to notice things like double[pyarrow] are not the same as numpy data types. Specifically, I didn't have trouble with them until I put them into my plotting routines (matplotlib) and then I got strange errors which took me a while to figure out were related to the data types.

Not sure what to do with this information, but thought it might be helpful for you to have the feedback. My solution is the following where I take just the columns I want from the lsdb returned: matched_df and converted them to np data types. I am sure there are other ways of handling this that might be more elegant, but this is one functional way.

df_lc = pd.DataFrame({
'flux': pd.to_numeric(matched_df['psfFlux'] * 1e3, errors='coerce').astype(np.float64),
'err': pd.to_numeric(matched_df['psfFluxErr'] * 1e3, errors='coerce').astype(np.float64),
'time': pd.to_numeric(matched_df['obsTime'], errors='coerce').astype(np.float64),
'objectid': matched_df['objectid'].astype(np.int64),
'band': filtername,
'label': matched_df['label'].astype(str)
}).set_index(["objectid", "label", "band", "time"])

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no repository files or tests. Start by reproducing a cross-match or join with Pan-STARRS catalogs and inspect the returned dataframe dtypes before passing data to matplotlib; completion would require an agreed way to handle or expose these types, which the issue does not define.

Written by the indexing model from the issue text.

Assessment

Tech stack
matplotlib, numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.