astropy / astropy/specutils

Which (FITS) header should be stored in Spectrum1D.meta (and where)?

Open
#617 17 comments 0 reactions 0 assignees View on GitHub
discussion io
Dominant language
Python
Stars
204
Forks
134
Avg merge
12h 55m
Merged PRs (30d)
1

Description

Loaders for FITS files are saving the full information from the header cards in `Spectrum1D.meta['header']`. In https://github.com/astropy/specutils/pull/608#discussion_r395632586 I noted some ambiguity as to *which* header should be stored.
Most of the `default_loaders` for such formats, namely `apogee`, `hst_*`, `muscles`, `subaru_pfs` and `sdss` `spec_loader`, are pulling the info from the FITS header of the first HDU, while the spectral data are read from one (or several) of the extension HDUs.
This primary HDU header will usually contain general information on the target, observing run etc., while the HDU with the spectrum will include more specific info on the dataset. In principle both could be of interest. Should there be a general recommendation for new default (or custom) loaders (and a fix to existing ones) how to handle this?

I the following options:

1. Leave it at the primary HDU header as with the majority of current `default_loaders`.
Pros: no API changes
Cons: discarding potentially useful information; inconsistent with Astropy's `io.fits` behaviour for `Table` (see below)

2. Always read the header for the HDU containing the data.
This is already implicitly done by `generic_spectrum_from_table` and `spectrum_from_column_mapping` as they are loading `table.meta` under the hood.
Pros: Matches the `Table.meta` data; possibility to use `io.fits` mechanism to filter relevant keywords (i.e. excluding `REMOVE_KEYWORDS` and all, that have already been used in defining column formats and units).
Cons: Losing info from the primary HDU header (among the current formats with data in a BINTABLE extension, at least the Apogee, HST, MUSCLES and SDSS also have potentially interesting information in the primary HDU).
Some formats, e.g. `apogee`, are parsing spectrum data from several different extensions.

3. Combine all header info from primary HDU and any extension HDU from which data are read.
This is already implemented in the `jwst_reader` by using the (FITS) `header.extend` method to update the primary header with the data extension header.
Pros: No or minimal loss of information.
Cons: Possibly excessive accumulation of values in `spectrum.meta['header']`; writing the spectrum back will produce a FITS file with different headers (but currently there is no guarantee that any of the loaders will write back identically formatted files anyway).

I tend to the 3rd option, using a similar implementation to `jwst_reader`. It might still be preferable to use the Astropy `Table.meta` dict for updating to take advantage of the `io.fits` mechanism for stripping excess cards.

**Addendum**
In the above I was assuming that the `Spectrum1D` format always has the header info collected in *one* dictionary within the meta dictionary as `Spectrum1D.meta['header']`; this is also suggested in the docs for creating a custom loader by prescribing
`meta = {'header': header}`
and followed by most of the `default_loaders`.
But in fact neither `parsing_utils` nor `jwst_reader` follow this scheme, but instead put all saved header cards directly into `Spectrum1D.meta`.
I therefore suggest to first settle on a consistent scheme for organising the `meta` dict.

Contributor guide

Open the contributing guide

Research direction

Start with parsing_utils, jwst_reader, generic_spectrum_from_table, spectrum_from_column_mapping, and the named default_loaders to compare how FITS headers enter Spectrum1D.meta. Review the linked PR discussion and determine a single metadata/header policy, then document the decision and bring affected loaders into line; done means the policy is unambiguous and existing loaders follow it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.