Spectra storage format specification

I would like to use this as a discussion point for fleshing out a data format spec for use in Starfish. This also pertains to #21. Spurred on by a recent post heard via @gully, I'd like to devote some time to ironing out this spec sooner rather than later. First, the possible choices (feel free to suggest more):

FITS: While astropy is already a dependency of Starfish, I would like to avoid using the FITS format to store spectra for reading into the code. This is primarily because it seems as though each observatory has a slightly different standard by which they store their data. For many of my spectra, I have been unable to use astropy to read the data reliably and in the end resorted to using Pyraf to convert the FITS file to text.

ASCSII: ASCII text files might be a good, simple solution that is easily understood by all and provides a low barrier for users ingesting spectra from new instruments. However, I do worry if future larger datasets may make this format unweildy. In particular, it isn't immediately clear to me how we should store echelle data in this format without requiring a separate file for each order.

NPY: Numpy save files. While this certainly would be easy from a Python standpoint, I would like to preserve some degree of cross-language compatibility in the data format, since there are many other routines in other languages that interact with spectra and it would be nice to allow them access. Also, preserving any metadata associated with the spectrum (e.g., barycentric correction) may be difficult.

HDF5: This format is my current preferred format. For the TRES spectra that I have been primarily dealing with, I have the good fortune that all orders contain the same number of pixels and so I have been storing spectra as 2D arrays with shape (norders, npix). As @gully mentioned in #21, this is unlikely to be the case for all spectrographs. I think I would prefer to move to a ragged array format, whereby there is a separate /00, /01, /02, etc folder for each order, within which is contained 1D arrays of wl, fl, and sigma. The order names need not necessarily be 0, 1, 2, if there are only a few orders it could also be something more descriptive like blue and red. HDF5 has the added benefit that we can continue to add in metadata as we see fit.

Masking: on a related note, when should we apply masks to data? If a raw spectrum from the telescope uses NaN or 1e-99 or other numbers to denote bad pixels, should we be propagating these values all the way through the fitting routines? Is it possible to clean these values from the spectrum as we are putting the raw spectrum into one of the above mentioned formats? My hope is that the user could clean these values in regions where it may be obvious (say 200 NaNs in a row at the edge of an order), but also allow a (possibly separate) mechanism by which the user can mask out real regions of the spectrum that cannot be reproduced by the model (e.g., telluric lines, cores of deep absorption lines). This second masking mechanism would likely be iterative, since this is more of a fitting task than a data task.

My current thinking is to continue using HDF5 in the hope that we can make the file format straightforward and easy to maintain, but I am interested in hearing what all of the contributors think!

No files, tests, or entry points are named. Start by resolving the storage-format and masking questions in this discussion, including ragged orders and metadata; done means the contributors agree on a complete, maintainable specification.