desihub / desihub/LSS

QSO maker broken for daily

Open
#212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
22
Forks
39
PR merge metrics
No merged PRs in 30d

Description

The QSO catalog maker uses pandas and pieces of it now fail due to numpy changes:

Traceback (most recent call last):
File "/global/u2/d/desica/LSScode/LSS/scripts/main/combdata_rel.py", line 254, in
ct.combtile_qso(tiles4comb, outf, restart=args.redoqso)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/main/cattools.py", line 73, in combtile_qso
tspec = combQSOdata(tile,zdate,tdate,cols=kl)
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/main/cattools.py", line 873, in combQSOdata
qso_cati = Table.from_pandas(qso_catalog_maker(rr, mgii, qn, old_extname_redrock, old_extname_for_qn, update_qn_zwarn = False))
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/qso_cat_utils.py", line 352, in qso_catalog_maker
zbest = read_fits_to_pandas(redrock, ext='ZBEST' if use_old_extname_for_redrock else 'REDSHIFTS', columns=columns_zbest)
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/qso_cat_utils.py", line 99, in read_fits_to_pandas
return pd.DataFrame(file.read().byteswap().newbyteorder())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

This is probably a good opportunity to switch to using what is coded up in desispec for selecting good redshifts and use that on line 873 in main/cattools.py

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in LSS/qso_cat_utils.py at read_fits_to_pandas (line 99) and reproduce the daily QSO catalog failure involving NumPy 2.0. Then inspect main/cattools.py at combQSOdata (line 873) and the desispec redshift-selection approach mentioned in the issue. Done means the daily QSO catalog maker runs without this error and applies the intended good-redshift selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.