QSO maker broken for daily
- Dominant language
- Jupyter Notebook
- Stars
- 22
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
The QSO catalog maker uses pandas and pieces of it now fail due to numpy changes:
Traceback (most recent call last):
File "/global/u2/d/desica/LSScode/LSS/scripts/main/combdata_rel.py", line 254, in
ct.combtile_qso(tiles4comb, outf, restart=args.redoqso)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/main/cattools.py", line 73, in combtile_qso
tspec = combQSOdata(tile,zdate,tdate,cols=kl)
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/main/cattools.py", line 873, in combQSOdata
qso_cati = Table.from_pandas(qso_catalog_maker(rr, mgii, qn, old_extname_redrock, old_extname_for_qn, update_qn_zwarn = False))
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/qso_cat_utils.py", line 352, in qso_catalog_maker
zbest = read_fits_to_pandas(redrock, ext='ZBEST' if use_old_extname_for_redrock else 'REDSHIFTS', columns=columns_zbest)
File "/global/common/software/desi/perlmutter/desiconda/20260227-2.3.1/code/LSS/main/py/LSS/qso_cat_utils.py", line 99, in read_fits_to_pandas
return pd.DataFrame(file.read().byteswap().newbyteorder())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.
This is probably a good opportunity to switch to using what is coded up in desispec for selecting good redshifts and use that on line 873 in main/cattools.py
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in LSS/qso_cat_utils.py at read_fits_to_pandas (line 99) and reproduce the daily QSO catalog failure involving NumPy 2.0. Then inspect main/cattools.py at combQSOdata (line 873) and the desispec redshift-selection approach mentioned in the issue. Done means the daily QSO catalog maker runs without this error and applies the intended good-redshift selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100