Enumerating all mailing lists is intensive in both CPU and network
- Dominant language
- Python
- Stars
- 45
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
In addition to the issue with list discovery identified in PR #16, there is an additional issue that the use of `size=0` later on in the query code to avoid `sum_other_doc_count` being greater than zero is strongly recommended against in the Elasticsearch documentation:
> It is possible to not limit the number of terms that are returned by setting `size` to `0`. Don’t use this on high-cardinality fields as this will kill both your CPU since terms need to be return sorted, and your network.
This means that the query will likely be very expensive on databases containing hundreds of thousands of messages, and `background.py` is running it once every couple of minutes or so. But it is necessary to use `size=0` in order to accurately enumerate all mailing lists.
The underlying issue here is that Elasticsearch is not designed for accurate queries of this nature over extremely large datasets. It may therefore be necessary to add an extra index for mailing lists, which would be updated whenever `archiver.py` receives another message.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading background.py and archiver.py, then review the mailing-list discovery change in PR #16 and the Elasticsearch guidance on high-cardinality terms queries. Determine whether a separate mailing-list index can be updated when archiver.py receives a message, and define how accurate enumeration and acceptable query cost will be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100