hypothesis / hypothesis/h

DataError: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type json

Open
#5,839 1 comment 0 reactions 0 assignees View on GitHub
bug S4 Sentry 💥
Dominant language
Python
Stars
3.2k
Forks
459
Avg merge
27d 1h
Merged PRs (30d)
1

Description

Creating an annotation can fail if an API client submits an annotation where one of the string values extracted from the document includes an incomplete UTF-16 surrogate pair (ie. first or second part missing).

One way this can be reproduced with the current client is to go to and select _exactly_ the text "上式中𝛿𝑥𝑎" which appears just below figure 1 and click "Annotate", then try to save the annotation. Instead of succeeding, it results in a 500 server error.

The submitted annotation includes the following data:

```
{"type":"'TextQuoteSelector'","prefix":"'\\udc3b\ud835\udeff\ud835\udc65 = \ud835\udc4f\u53ef\u5199\u6210\u5982\u4e0b\u5f62\u5f0f:\\n\uff081\uff09\\n\\xa0\\n\u503c\u5f97\u8bf4\u660e\u7684\u662f\uff0c'","exact":"'\u4e0a\u5f0f\u4e2d\ud835\udeff\ud835\udc65\ud835\udc4e\u3001\ud835\udeff\ud835\udc65\ud835\udc4f\u5e76\u975e\u4e00\u5b9a\u4e3a\u76f8\u673a\u4f4d\u59ff\u90e8\u5206\u548c\u8def\u6807\u90e8\u5206\uff0c\u800c\u662f\u5e0c\u671b marg \u7684\u90e8\u5206\u548c\u5e0c\u671b\u4fdd\u7559\u7684\u90e8\u5206'","suffix":"'\u3002\u53e6\u5916\uff0cVINS \u4e2d\u7684\u8fb9\u7f18\u5316\u4e0e G2O \u8ba1\u7b97\u8fc7\u7a0b\u4e2d\u7684\u8fb9\u7f18\u5316\u610f\u4e49\u4e0d'"}
```

Note that the `prefix` starts with `\udc3b` which is [not a valid unicode character](https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF). I believe the reason this is happening is because the library that generates the `TextQuoteSelector` uses a fixed number of UTF-16 characters as the prefix (32 AFAIK). If you select _just_ the right text (which becomes the _exact_ field) in _just_ the right context, then the generated prefix can end up chopping a UTF-16 surrogate pair in half.

It would be reasonable in this context for the server to either fix the invalid data somehow (eg. by removing invalid UTF chars) or to respond with a 400 error and require a fix in the client. I will note that this is quite an easy mistake for a JavaScript-based API client to make because strings use UTF-16 chars and indexing works on UTF-16 code units rather than whole characters.

----

Sentry Issue: [H-1VF](https://sentry.io/organizations/hypothesis/issues/1342576233/?referrer=github_integration)

```
InvalidTextRepresentation: invalid input syntax for type json
LINE 2: .../www.cnblogs.com/feifanrensheng/p/10532918.html', '[{"type":...
^
DETAIL: Unicode low surrogate must follow a high surrogate.
CONTEXT: JSON data, line 1: ...d\ud835\udeff\ud835\udc65\ud835\udc4e", "prefix":...

File "sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
File "newrelic/hooks/database_psycopg2.py", line 51, in execute
**kwargs)
File "newrelic/hooks/database_dbapi2.py", line 25, in execute
*args, **kwargs)

DataError: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type json
LINE 2: .../www.cnblogs.com/feifanrensheng/p/10532918.html', '[{"type":...
^
DETAIL: Unicode low surrogate must follow a high surrogate.
CONTEXT: JSON data, line 1: ...d\ud835\udeff\ud835\udc65\ud835\udc4e", "prefix":...

[SQL: INSERT INTO annotation (created, updated, userid, groupid, text, text_rendered, tags, shared, target_uri, target_uri_normalized, target_sel...
(36 additional frame(s) were not displayed)
...
File "sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
File "newrelic/hooks/database_psycopg2.py", line 51, in execute
**kwargs)
File "newrelic/hooks/database_dbapi2.py", line 25, in execute
*args, **kwargs)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.