Indexing search failures
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 459
- Avg merge
- 27d 1h
- Merged PRs (30d)
- 1
Description
Encountered during a rebuild from a production dump, 37 of our annotations failed to reindex.
The [list of failing IDs](https://hypothes-is.slack.com/files/dthompson/F49GQRWQ7/reindexing_failures.txt) is available to those with Slack access.
From a brief investigation, these all appear to be instances of the same bug. I ran the following code:
```python
from memex.search.indexer import index
annotation_ids = set([ … ])
for a_id in annotation_ids:
annotation = session.query(models.Annotation).get(a_id)
try:
index(request.es, annotation, request)
except Exception as e:
print e
```
…and I got a series of exceptions, all of the form:
`TransportError(500, u'IllegalArgumentException[Document contains at least one immense term in field="target.selector.exact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: \'[85, 112, 32, 78, 101, 120, 116, 67, 97, 110, 99, 101, 108, 65, 117, 116, 111, 112, 108, 97, 121, 32, 105, 115, 32, 112, 97, 117, 115, 101]...\', original message: bytes can be at most 32766 in length; got 70229]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 70229]; ')`
The prefix it talks about maps to:
```python
In [4]: ''.join([chr(x) for x in prefix])
Out[4]: 'Up NextCancelAutoplay is pause'
```
…which I hope means more to someone else than it does to me.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.