hypothesis / hypothesis/h

Indexing search failures

Open
#4,387 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3.2k
Forks
459
Avg merge
27d 1h
Merged PRs (30d)
1

Description

Encountered during a rebuild from a production dump, 37 of our annotations failed to reindex.

The [list of failing IDs](https://hypothes-is.slack.com/files/dthompson/F49GQRWQ7/reindexing_failures.txt) is available to those with Slack access.

From a brief investigation, these all appear to be instances of the same bug. I ran the following code:

```python
from memex.search.indexer import index

annotation_ids = set([ … ])
for a_id in annotation_ids:
annotation = session.query(models.Annotation).get(a_id)
try:
index(request.es, annotation, request)
except Exception as e:
print e
```

…and I got a series of exceptions, all of the form:

`TransportError(500, u'IllegalArgumentException[Document contains at least one immense term in field="target.selector.exact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: \'[85, 112, 32, 78, 101, 120, 116, 67, 97, 110, 99, 101, 108, 65, 117, 116, 111, 112, 108, 97, 121, 32, 105, 115, 32, 112, 97, 117, 115, 101]...\', original message: bytes can be at most 32766 in length; got 70229]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 70229]; ')`

The prefix it talks about maps to:

```python
In [4]: ''.join([chr(x) for x in prefix])
Out[4]: 'Up NextCancelAutoplay is pause'
```

…which I hope means more to someone else than it does to me.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.