lincc-frameworks / lincc-frameworks/hyrax
Protect Unsafe `to_pandas` conversion
@aritraghsh09 is already working on this.
Since Sep 18, 2025.
- Dominant language
- Python
- Stars
- 41
- Forks
- 7
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 8
Description
The conversion of an astropy table to a pandas dataframe using `to_pandas` is not safe if there are masked ints anywhere in the table; as these ints are coerced into floats; causing them to possibly loose precision.
Protect all instances of `to_pandas` at the very least with something along the lines of
```{python}
#################
# when using to_pandas, if there are masked (nan)
# values in an int column; then the conversion is
# forced through a float. This causes predicision issues
#
# p.any(np.abs(unmasked_data) > 2**53) is correct
# we do lower threshold for added safety
##################
def quick_precision_check(table, names=None):
if names is None:
names = table.colnames
risky_cols = []
for col_name in names:
col = table[col_name]
if (col.dtype.kind in ['i', 'u'] and
hasattr(col, 'mask') and col.mask is not np.ma.nomask and
col.mask.any()):
unmasked_data = col.data[~col.mask]
if len(unmasked_data) > 0 and np.any(np.abs(unmasked_data) > 2**40):
risky_cols.append(col_name)
return risky_cols
```
As of right now, the only instance of `to_pandas` I found in hyrax was in [src/hyrax/3d_viz/save_umap_to_json.py](https://github.com/lincc-frameworks/hyrax/blob/93366bf6de9890504b2025263e20bdd2f3aceb63/src/hyrax/3d_viz/save_umap_to_json.py#L255)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.