apache / apache/ponymail-foal

Enumerating all mailing lists is intensive in both CPU and network

Open
#17 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
45
Forks
23
PR merge metrics
No merged PRs in 30d

Description

In addition to the issue with list discovery identified in PR #16, there is an additional issue that the use of `size=0` later on in the query code to avoid `sum_other_doc_count` being greater than zero is strongly recommended against in the Elasticsearch documentation:

> It is possible to not limit the number of terms that are returned by setting `size` to `0`. Don’t use this on high-cardinality fields as this will kill both your CPU since terms need to be return sorted, and your network.

This means that the query will likely be very expensive on databases containing hundreds of thousands of messages, and `background.py` is running it once every couple of minutes or so. But it is necessary to use `size=0` in order to accurately enumerate all mailing lists.

The underlying issue here is that Elasticsearch is not designed for accurate queries of this nature over extremely large datasets. It may therefore be necessary to add an extra index for mailing lists, which would be updated whenever `archiver.py` receives another message.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading background.py and archiver.py, then review the mailing-list discovery change in PR #16 and the Elasticsearch guidance on high-cardinality terms queries. Determine whether a separate mailing-list index can be updated when archiver.py receives a message, and define how accurate enumeration and acceptable query cost will be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.