votable parsing issue: datatype="char" casted as object dtype for Table
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 2.2k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 74
Description
I downloaded a table from the Gaia science archive using astroquery, and tried to write it to a FITS file:
```python
from astroquery.gaia import Gaia
query = '''SELECT TOP 10 * FROM gaiadr1.allwise_original_valid'''
job = Gaia.launch_job_async(query)
res = job.get_results()
res.write('/tmp/test.fits')
```
but get an error:
```
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
in ()
2 job = Gaia.launch_job_async(query)
3 res = job.get_results()
----> 4 res.write('/tmp/test.fits')
~/anaconda/lib/python3.6/site-packages/astropy/table/table.py in write(self, *args, **kwargs)
2558 passed through to the underlying data reader (e.g. `~astropy.io.ascii.write`).
2559 """
-> 2560 io_registry.write(self, *args, **kwargs)
2561
2562 def copy(self, copy_data=True):
~/anaconda/lib/python3.6/site-packages/astropy/io/registry.py in write(data, format, *args, **kwargs)
558
559 writer = get_writer(format, data.__class__)
--> 560 writer(data, *args, **kwargs)
561
562
~/anaconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in write_table_fits(input, output, overwrite)
376
377 # Encode any mixin columns into standard Columns.
--> 378 input = _encode_mixins(input)
379
380 table_hdu = table_to_hdu(input, character_as_bytes=True)
~/anaconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in _encode_mixins(tbl)
353 idxs = list(range(0, len(line) + 70, 70))
354 lines = [line[i0:i1] + '\\' for i0, i1 in zip(idxs[:-1], idxs[1:])]
--> 355 lines[-1] = lines[-1][:-1]
356 encode_tbl.meta['comments'].extend(lines)
357
IndexError: list index out of range
```
Is this a bug in the way the table is constructed from the results in astroquery? Either way, I think the error message could be much more informative.
I see the word mixin, so I tried converting all columns to plain numpy arrays, and that does seem to work:
```python
for c in tbl.colnames:
tbl[c] = np.array(tbl[c])
tbl.write('/tmp/test.fits')
```
This is somewhere between a bug in astroquery and in astropy, so I've opened an issue in both. (See astropy/astroquery#1145)
Contributor guide
Research direction
Reproduce the Gaia query and FITS write shown in the issue, then inspect astropy/io/fits/connect.py around _encode_mixins and table_to_hdu, along with astropy/table/table.py. Confirm how the char column becomes object dtype and ensure the same table can be written without the reported IndexError or produces a more informative error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100