ENH: Better support for ESO decompression (particularly on Windows)
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
This is a follow up to issue #1818 and its corresponding PR #2681.
### Issue
Currently, the implementation of `eso.core._unzip_file` handles `.Z` and `.gz` files by doing system calls to `gunzip`:
https://github.com/astropy/astroquery/blob/d927bdfd8478ca480f919dd7938b4a7d8797537a/astroquery/eso/core.py#L681-L697
While most (though not necessarily all) Linux/MacOS users are able to do this system call, Windows users are unlikely to have `gunzip` installed, especially in a system-wide fashion. In other words, the current code is incompatible with most Windows users. Another downside is that the current code creates a full decompressed copy of the file in storage, which may be unnecessary in cases where we only need to read a small part of the file (not to mention storage space concerns).
Right, now if one attempts to perform the following code on a Windows install with no gunzip:
```python
from astroquery.eso import Eso
files = Eso().retrieve_data("NACO.2016-09-16T08:37:27.966", destination="test_folder")
```
it prints out the error:
```
UserWarning: Unable to unzip files (gunzip is not available on this system)
```
And no files are made available on the folder, not even the compressed files.
### Possible solution
For `.gz` files, the alternative is clear: we can simply use Python's `gzip` module, which will work equally on all OSes. This option also enables us to support decompression on-the-fly, avoiding the full decompressed copy. It should also perform similarly to the system call, possibly better as there is less overhead in principle.
As for `.Z` files, they are a bit trickier. We went through a similar issue in Astropy (see astropy/astropy#10714), because the `gzip` module does not support them and thus there is no built-in module in Python for handling `.Z` files. To solve this issue, I ended up creating [`uncompresspy`](https://github.com/kYwzor/uncompresspy/) which is a Pure Python package for reading .Z files that supports decompression on-the-fly and works on all OSes. This option was recently added to Astropy (see astropy/astropy#17960) and so far it has worked fine with all known files. There are some performance concerns (see https://github.com/astropy/astropy/issues/10714#issuecomment-2766649330), but I believe integrating it in astroquery would be possible and an improvement at least for all users that do not have `gunzip`. A possible concern is that `uncompresspy` only supports Python 3.10+, while Astroquery provides support for Python 3.9+, but I'm not sure how important this is, given that the official support for Python 3.9 will be dropped in just a few days and uncompresspy would always be an optional dependency.
More generally, I wonder if the compression handling logic here could be replaced by Astropy's [`get_readable_fileobj`](https://docs.astropy.org/en/stable/api/astropy.utils.data.get_readable_fileobj.html)? Does the ESO archive ever store anything in a compressed format that is not `.gz` or `.Z`?
If there's interest in this, I could propose a PR, but I'd need some guidance on how you'd want me to approach this (e.g. should we just prefer `gunzip` system calls for those that do have it, and fallback to `gzip`/`uncompresspy `for those that don't?)
Contributor guide
Research direction
Start at astroquery/eso/core.py, especially eso.core._unzip_file at lines 681-697, and trace Eso().retrieve_data(...). Review Astropy's get_readable_fileobj and the gzip/uncompresspy options described here, including Python-version compatibility. Done means ESO .gz/.Z retrieval works without a system gunzip on Windows and does not discard compressed files when decompression is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100