Smithsonian / Smithsonian/layup
obs80 reader: a radar record aborts the whole file instead of being skipped and counted
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10
- Forks
- 2
- Avg merge
- 1d 43m
- Merged PRs (30d)
- 58
Description
A matched radar (R/r) two-line record aborts the whole read, taking every other object in the file with it. On a four-line file -- two ordinary observations plus one radar pair -- read_rows() raises and returns nothing:
ValueError: Radar (R/r) obs80 two-line records are not supported by
Obs80DataReader (object 00009); ingest radar via ADES delay/doppler columns.
The refusal itself is right and the message is clear. The problem is that it is fatal to the file rather than to the record. Obs80DataReader has no try/except and no logging anywhere, so there is no path that skips a record it cannot represent, and no count of what was skipped. A user with one radar observation in an archive extract loses the archive.
Two things would help:
- skip a record the reader deliberately refuses, rather than raising, and keep genuine corruption (a short line, a mismatched observatory code) fatal as it is now;
- report how many records were skipped and of what type, so a partial loss is visible. Today a silent partial loss is indistinguishable from a clean read.
The natural place for both is _iter_records, which already drops orphaned and mismatched continuation lines silently. Doing it there also keeps get_row_count() consistent with what read_rows() actually returns.
Correction, same day. This issue as first filed was written against a stale checkout and was mostly wrong. For the record, on current main these claims of mine do not hold:
- "treats every second line as a satellite position" -- false.
convert_obs80dispatches on the first line's note 2 (ec98eda). - "a roving-observer record aborts the read" -- false. Roving records read correctly to
sys = WGS84with geodetic longitude, latitude and altitude. - "an East longitude near 2xx degrees is accepted as ICRF_AU" -- false. That branch is reached only for
S. My test fixture was missing the1that a real roving continuation line carries in column 33, and the failure was in my fixture, not the code.
Apologies for the noise. Only the radar abort and the missing skip-and-count survive, and the issue above is rewritten to those.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Obs80DataReader._iter_records and trace how read_rows() and get_row_count() consume its records; review convert_obs80 for the radar refusal. Verify that a file containing ordinary observations and a radar pair returns the ordinary records, reports the skipped radar count and type, and still raises for short lines or mismatched observatory codes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100