UnicodeDecodeError, ascii codec can't decode byte
- Dominant language
- Python
- Stars
- 88
- Forks
- 64
- Avg merge
- 38m
- Merged PRs (30d)
- 1
Description
**OS**: Mac OS 12.5
**Python**: 3.9.13
**PyVO**: 1.3
PyVO raises an exception with the following query:
``` py
import pyvo
service = pyvo.dal.TAPService("https://gea.esac.esa.int/tap-server/tap")
results = service.search(
" ".join((
"SELECT table_name, description",
"FROM tap_schema.tables",
"WHERE table_name = 'gaiadr2.dr1_neighbourhood'"
))
)
```
The exception:
```
Traceback (most recent call last):
File "/tmp/pyvo-encoding.py", line 4, in
results = service.search(
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/tap.py", line 246, in run_sync
return self.create_query(
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/tap.py", line 942, in execute
return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
File "/usr/local/lib/python3.9/site-packages/pyvo/dal/query.py", line 245, in execute_votable
raise DALFormatError(e, self.queryurl)
pyvo.dal.exceptions.DALFormatError: UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 1317: ordinal not in range(128)
```
If I query the same thing with a bare cURL:
``` sh
$ curl -X "POST" "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=JSON&QUERY=SELECT%20table_name,%20description%20FROM%20tap_schema.tables%20WHERE%20table_name%20=%20%27gaiadr2.dr1_neighbourhood%27"
```
then the result is the following:
``` json
{
"metadata":
[
{
"name": "table_name",
"datatype": "char",
"xtype": null,
"arraysize": "*",
"description": null,
"unit": null,
"ucd": null,
"utype": null
},
{
"name": "description",
"datatype": "char",
"xtype": null,
"arraysize": "*",
"description": null,
"unit": null,
"ucd": null,
"utype": null
}
],
"data":
[
[
"gaiadr2.dr1_neighbourhood",
"Users wishing to look up the DR2 record for an astrophysical source\nidentified in DR1 must NOT simply extract the record from DR2 having the\nsame source identifier.\n\nAs described in the detailed description of attribute designation in\nGaiaSource it is not guaranteed that the same astronomical source will\nalways have the same source identifier in different Data Releases. Hence\nthe only safe way to compare source records between different Data\nReleases in general is to check the records of proximal source(s) in the\nsame small part of the sky. This table provides the means to do this via\na precomputed crossmatch of such sources, taking into account the proper\nmotions available at DR2.\n\nWithin the neighbourhood of a given DR2 source there may be none, one or\n(rarely) several possible counterparts in DR1 indicated by rows in this\ntable. This occasional source confusion was introduced during the DR1\nprocessing which used an earlier version of the software for matching of\ntransit observations to unique astrophysical sources. The subsequent\nmerging, splitting and deletion of identifiers introduced at DR1 during\nthe DR2 processing means there is no guaranteed one–to–one\ncorrespondence in source identifiers between the releases.\n\nFor more details of the procedure used to create this crossmatch, see\nSection [chap:xmdr1] in the online documentation."
]
]
}
```
You can't see it here, as the symbol is kind of invisible, but the problematic place does have this `<0xa0>` symbol in `Section<0xa0>[chap:xmdr1]`, which I guess is what is causing the problem.
If TAP specification dictates all the services to return results in ASCII only, then I'd say it's certainly the Gaia service fault and not PyVO, but even then I'd say it would be useful to be able to specify the encoding for reading the results (*I'm assuming that the same string would read fine with UTF-8*), as right now it seems to be "hardcoded" to ASCII.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pyvo/dal/query.py at execute_votable and trace the response decoding through pyvo/dal/tap.py. Reproduce the Gaia TAP query and inspect how the non-ASCII byte in the JSON response is decoded; done means the query no longer raises DALFormatError and the response parses successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100