hypothesis / hypothesis/h

Oversized annotation payloads cause Elasticsearch to explode

Open
#2,931 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3.2k
Forks
459
Avg merge
27d 1h
Merged PRs (30d)
1

Description

Absurdly large annotation payloads (most of the below looks like errors in scibot) cause 500 Errors in Elasticsearch.

One way to address some issues here is to impose a reasonable maximum size on the overall annotation payload and reject requests that exceed that size with a 400 error (or a 413 error if we're being fancy).

Another option is to impose maximum sizes on specific individual fields such as URL or selector data. For example, a URL over 2K characters is unlikely to work in most browsers, so rejecting URLs over 5K would seem reasonable for us.

https://app.getsentry.com/hypothesis/prod/issues/106485449/

```
TransportError: TransportError(500, u'IllegalArgumentException[Document contains at least one immense term in field="target.selector.exact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: \'[10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 10, 32, 32, 32, 32, 32, 32, 32, 32]...\', original message: bytes can be at most 32766 in length; got 51481]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 51481]; ')
(16 additional frame(s) were not displayed)
...
File "elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "elasticsearch/client/__init__.py", line 263, in index
_make_path(index, doc_type, id), params=params, body=body)
File "elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "elasticsearch/connection/http_urllib3.py", line 93, in perform_request
self._raise_error(response.status, raw_data)
File "elasticsearch/connection/base.py", line 105, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.