LocalFileSystem incompatibility / corruption when used to read FITS files

When opening the same local file via open() and via fsspec.open() I get garbled floating point values from the latter when the library loads it. Confusingly, the ascii header of the file seems to come through fine, so it's at least partially loading the file... (I have implemented an iRODS interface through fsspec, and don't see the issue when opening files from iRODS, so as far as I know this is specific to local paths with fsspec.)

When read through fsspec instead of regular open(), the astropy.io.fits.open function is able to parse the header (a sort of ascii preamble) but the associated data are garbled. I'm trying to discourage astropy.io.fits from trying to memory map or seek or anything (for compatibility with other fsspec backends), hence the mode='readonly', memmap=False in the fits.open call.

I'm totally stumped, but I can reproduce with a short script, included below. It's possible this is an issue that Astropy needs to fix, but I can't figure out what would conceivably be different about the file proxy object that would let it read the header and not the data...

To reproduce:

1. Save this script as repro.py

import numpy as np
import fsspec
from astropy.io import fits

def main():
    data = np.arange(10, dtype=float)
    fits.PrimaryHDU(data).writeto('ex.fits', overwrite=True)
    print('\nUsing builtin open()')
    with open('ex.fits', 'rb') as fh:
        hdul = fits.open(fh, mode='readonly', memmap=False)
        hdul.info()
        print(hdul[0].data)

    print('\nUsing fsspec.open() with local path')
    with fsspec.open('ex.fits', 'rb') as fh:
        hdul = fits.open(fh, mode='readonly', memmap=False)
        hdul.info()
        print(hdul[0].data)

if __name__ == "__main__":
    main()

2. Create a virtual env

python -m venv env
source env/bin/activate
pip install astropy fsspec

3. Compare output from two approaches to open ex.fits, e.g.

$ python repro.py

Using builtin open()
Filename: ex.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU       5   (10,)   float64
[0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]

Using fsspec.open() with local path
Filename: /Users/josephlong/devel/repro/ex.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU       5   (10,)   float64
[1.64931512e+093 2.86446248e-014 6.01347002e-154 6.01347002e-154
 1.15963942e-152 3.59533712e+246 2.68090142e+092 1.07357647e+243
 6.01347002e-154 6.01347002e-154]

Here's the same from my own test with iRODS / the irods_fsspec backend

Using irods_fsspec through fsspec
Filename: <class '_io.BufferedRandom'>
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU       5   (10,)   float64
[0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]

Start by running the supplied repro.py and compare the two calls to fsspec.open() and fits.open(). Trace the local filesystem file object used by fsspec against the builtin open() path, focusing on why headers are correct but data reads are corrupted. Done means the local fsspec path returns the same FITS data as builtin open() without breaking the iRODS behavior.