astropy / astropy/pyvo

UnicodeDecodeError, ascii codec can't decode byte

Open
#346 4 comments 0 reactions 0 assignees View on GitHub
bug upstream
Dominant language
Python
Stars
88
Forks
64
Avg merge
38m
Merged PRs (30d)
1

Description

**OS**: Mac OS 12.5
**Python**: 3.9.13
**PyVO**: 1.3

PyVO raises an exception with the following query:

``` py
import pyvo

service = pyvo.dal.TAPService("https://gea.esac.esa.int/tap-server/tap")
results = service.search(
" ".join((
"SELECT table_name, description",
"FROM tap_schema.tables",
"WHERE table_name = 'gaiadr2.dr1_neighbourhood'"
))
)
```

The exception:

```
Traceback (most recent call last):
File "/tmp/pyvo-encoding.py", line 4, in
results = service.search(
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/tap.py", line 246, in run_sync
return self.create_query(
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/tap.py", line 942, in execute
return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/query.py", line 245, in execute_votable
raise DALFormatError(e, self.queryurl)
pyvo.dal.exceptions.DALFormatError: UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 1317: ordinal not in range(128)
```

If I query the same thing with a bare cURL:

``` sh
$ curl -X "POST" "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=JSON&QUERY=SELECT%20table_name,%20description%20FROM%20tap_schema.tables%20WHERE%20table_name%20=%20%27gaiadr2.dr1_neighbourhood%27"
```

then the result is the following:

``` json
{
"metadata":
[
{
"name": "table_name",
"datatype": "char",
"xtype": null,
"arraysize": "*",
"description": null,
"unit": null,
"ucd": null,
"utype": null
},
{
"name": "description",
"datatype": "char",
"xtype": null,
"arraysize": "*",
"description": null,
"unit": null,
"ucd": null,
"utype": null
}
],
"data":
[
[
"gaiadr2.dr1_neighbourhood",
"Users wishing to look up the DR2 record for an astrophysical source\nidentified in DR1 must NOT simply extract the record from DR2 having the\nsame source identifier.\n\nAs described in the detailed description of attribute designation in\nGaiaSource it is not guaranteed that the same astronomical source will\nalways have the same source identifier in different Data Releases. Hence\nthe only safe way to compare source records between different Data\nReleases in general is to check the records of proximal source(s) in the\nsame small part of the sky. This table provides the means to do this via\na precomputed crossmatch of such sources, taking into account the proper\nmotions available at DR2.\n\nWithin the neighbourhood of a given DR2 source there may be none, one or\n(rarely) several possible counterparts in DR1 indicated by rows in this\ntable. This occasional source confusion was introduced during the DR1\nprocessing which used an earlier version of the software for matching of\ntransit observations to unique astrophysical sources. The subsequent\nmerging, splitting and deletion of identifiers introduced at DR1 during\nthe DR2 processing means there is no guaranteed one–to–one\ncorrespondence in source identifiers between the releases.\n\nFor more details of the procedure used to create this crossmatch, see\nSection [chap:xmdr1] in the online documentation."
]
]
}
```

You can't see it here, as the symbol is kind of invisible, but the problematic place does have this `<0xa0>` symbol in `Section<0xa0>[chap:xmdr1]`, which I guess is what is causing the problem.

If TAP specification dictates all the services to return results in ASCII only, then I'd say it's certainly the Gaia service fault and not PyVO, but even then I'd say it would be useful to be able to specify the encoding for reading the results (*I'm assuming that the same string would read fine with UTF-8*), as right now it seems to be "hardcoded" to ASCII.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pyvo/dal/query.py at execute_votable and trace the response decoding through pyvo/dal/tap.py. Reproduce the Gaia TAP query and inspect how the non-ASCII byte in the JSON response is decoded; done means the query no longer raises DALFormatError and the response parses successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.