DataError: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type json
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 459
- Avg merge
- 27d 1h
- Merged PRs (30d)
- 1
Description
Creating an annotation can fail if an API client submits an annotation where one of the string values extracted from the document includes an incomplete UTF-16 surrogate pair (ie. first or second part missing).
One way this can be reproduced with the current client is to go to and select _exactly_ the text "上式中𝛿𝑥𝑎" which appears just below figure 1 and click "Annotate", then try to save the annotation. Instead of succeeding, it results in a 500 server error.
The submitted annotation includes the following data:
```
{"type":"'TextQuoteSelector'","prefix":"'\\udc3b\ud835\udeff\ud835\udc65 = \ud835\udc4f\u53ef\u5199\u6210\u5982\u4e0b\u5f62\u5f0f:\\n\uff081\uff09\\n\\xa0\\n\u503c\u5f97\u8bf4\u660e\u7684\u662f\uff0c'","exact":"'\u4e0a\u5f0f\u4e2d\ud835\udeff\ud835\udc65\ud835\udc4e\u3001\ud835\udeff\ud835\udc65\ud835\udc4f\u5e76\u975e\u4e00\u5b9a\u4e3a\u76f8\u673a\u4f4d\u59ff\u90e8\u5206\u548c\u8def\u6807\u90e8\u5206\uff0c\u800c\u662f\u5e0c\u671b marg \u7684\u90e8\u5206\u548c\u5e0c\u671b\u4fdd\u7559\u7684\u90e8\u5206'","suffix":"'\u3002\u53e6\u5916\uff0cVINS \u4e2d\u7684\u8fb9\u7f18\u5316\u4e0e G2O \u8ba1\u7b97\u8fc7\u7a0b\u4e2d\u7684\u8fb9\u7f18\u5316\u610f\u4e49\u4e0d'"}
```
Note that the `prefix` starts with `\udc3b` which is [not a valid unicode character](https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF). I believe the reason this is happening is because the library that generates the `TextQuoteSelector` uses a fixed number of UTF-16 characters as the prefix (32 AFAIK). If you select _just_ the right text (which becomes the _exact_ field) in _just_ the right context, then the generated prefix can end up chopping a UTF-16 surrogate pair in half.
It would be reasonable in this context for the server to either fix the invalid data somehow (eg. by removing invalid UTF chars) or to respond with a 400 error and require a fix in the client. I will note that this is quite an easy mistake for a JavaScript-based API client to make because strings use UTF-16 chars and indexing works on UTF-16 code units rather than whole characters.
----
Sentry Issue: [H-1VF](https://sentry.io/organizations/hypothesis/issues/1342576233/?referrer=github_integration)
```
InvalidTextRepresentation: invalid input syntax for type json
LINE 2: .../www.cnblogs.com/feifanrensheng/p/10532918.html', '[{"type":...
^
DETAIL: Unicode low surrogate must follow a high surrogate.
CONTEXT: JSON data, line 1: ...d\ud835\udeff\ud835\udc65\ud835\udc4e", "prefix":...
File "sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
File "newrelic/hooks/database_psycopg2.py", line 51, in execute
**kwargs)
File "newrelic/hooks/database_dbapi2.py", line 25, in execute
*args, **kwargs)
DataError: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type json
LINE 2: .../www.cnblogs.com/feifanrensheng/p/10532918.html', '[{"type":...
^
DETAIL: Unicode low surrogate must follow a high surrogate.
CONTEXT: JSON data, line 1: ...d\ud835\udeff\ud835\udc65\ud835\udc4e", "prefix":...
[SQL: INSERT INTO annotation (created, updated, userid, groupid, text, text_rendered, tags, shared, target_uri, target_uri_normalized, target_sel...
(36 additional frame(s) were not displayed)
...
File "sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
File "newrelic/hooks/database_psycopg2.py", line 51, in execute
**kwargs)
File "newrelic/hooks/database_dbapi2.py", line 25, in execute
*args, **kwargs)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.