Oversized annotation payloads cause Elasticsearch to explode
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 459
- Avg merge
- 27d 1h
- Merged PRs (30d)
- 1
Description
Absurdly large annotation payloads (most of the below looks like errors in scibot) cause 500 Errors in Elasticsearch.
One way to address some issues here is to impose a reasonable maximum size on the overall annotation payload and reject requests that exceed that size with a 400 error (or a 413 error if we're being fancy).
Another option is to impose maximum sizes on specific individual fields such as URL or selector data. For example, a URL over 2K characters is unlikely to work in most browsers, so rejecting URLs over 5K would seem reasonable for us.
https://app.getsentry.com/hypothesis/prod/issues/106485449/
```
TransportError: TransportError(500, u'IllegalArgumentException[Document contains at least one immense term in field="target.selector.exact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: \'[10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 10, 32, 32, 32, 32, 32, 32, 32, 32]...\', original message: bytes can be at most 32766 in length; got 51481]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 51481]; ')
(16 additional frame(s) were not displayed)
...
File "elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "elasticsearch/client/__init__.py", line 263, in index
_make_path(index, doc_type, id), params=params, body=body)
File "elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "elasticsearch/connection/http_urllib3.py", line 93, in perform_request
self._raise_error(response.status, raw_data)
File "elasticsearch/connection/base.py", line 105, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.