memmap exception if BLANK header is set
- Dominant language
- Python
- Stars
- 127
- Forks
- 74
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
Hi —
I'm trying to use reproject with an integer-valued image that has the BLANK keyword set. This turns out to lead to an exception, at least with the interpolating mode:
```
ValueError: mmap length is greater than file size
```
The problem is because of [`hdu_to_numpy_memmap()`][1]. It has various guards in place to only memmap if the HDU `data` array aligns exactly with what's on disk. The presence of BLANK causes Astropy to change my `u16` data to `f32`, which means that the memmap should not be performed, but as things currently stand, the check doesn't get this right.
[1]: https://github.com/astropy/reproject/blob/main/reproject/utils.py#L71
The current guard code is:
```py
if (
hdu.header.get("BSCALE", 1) != 1
or hdu.header.get("BZERO", 0) != 0
or hdu.fileinfo() is None
or hdu._data_replaced
or hdu.fileinfo()["file"].compression is not None
):
return hdu.data
```
I could easily add a test for the existence of the BLANK keyword, but I'm wondering if there's a "better" way to check for this case. Looking at the Astropy FITS code, there's a field called `_data_needs_rescale` alongside the existing `_data_replaced`; it's False for this HDU, but maybe that's an Astropy bug, and it should be true? There are also an `hdu._blank` field. Naively, maybe it's an Astropy bug that `_data_replaced` is not True in this situation?
I'm happy to submit a pull request for this, as well as provide a specific reproduction case, but I was wondering if anyone had input about the best approach to take.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in reproject/utils.py at hdu_to_numpy_memmap() and reproduce the ValueError with an integer-valued image whose FITS header contains BLANK. Inspect the existing memmap guards alongside Astropy's _data_needs_rescale, _data_replaced, and _blank fields. Done means the reproduction no longer attempts an invalid memmap and a regression test covers the BLANK case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100