Customize kiwix-serve cache settings to limit memory consumption
- Dominant language
- Python
- Stars
- 15
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
As of today, kiwix-serve cache settings are not customized on library.kiwix.org (and not on dev.library.kiwix.org)
As discussed in https://github.com/kiwix/libkiwix/issues/1025, kiwix-serve is using a significant amount of memory. With current code, we could probably put more control on this memory consumption by customizing some settings explained below
| Environment variable | Purpose | Default value | Comment |
|---|---|---|---|
| `KIWIX_ARCHIVE_CACHE_SIZE` | Number of open readers (~ZIM) | 10% of `getBookCount_not_protected` (number of local and remote books) ~= 421 today | |
| `KIWIX_SEARCHER_CACHE_SIZE` | Number of open searcher (which might include readers non accounted for in `KIWIX_ARCHIVE_CACHE_SIZE`) | idem `KIWIX_ARCHIVE_CACHE_SIZE` ~= 421 today | |
| `ZIM_DIRENTCACHE` | Number of dirent kept in cache per ZIM | 512 | Probably low impact on memory |
| `ZIM_DIRENTLOOKUPCACHE` | Idem `ZIM_DIRENTCACHE` | 1024 | Probably low impact on memory |
| `ZIM_CLUSTERCACHE` | Number of cluster kept in cache per ZIM | 16 | |
My gut feeling is that 412 for `KIWIX_ARCHIVE_CACHE_SIZE` and `KIWIX_SEARCHER_CACHE_SIZE` is way too much, I wouldn't assume we open this amount of ZIM every day, but my experience is limited.
I suggest that we do a small experiments directly in production on library.kiwix.org (dev.library.kiwix.org is not really pertinent in terms of number of ZIMs + traffic and has known issues):
- instead of having 2 kiwix-serve containers in deployment `library-data`, reduce this to 1
- create a new deployment `library-data-expe`, with 1 kiwix-serve container and custom environment variables
- modify `library-data` service to redirect to both k8s deployment
- this is easy to do (easier than at varnish side)
- should we encounter a problem, we just scale `library-data` to 2 containers and `library-data-expe` to 0 et voilà
- for every experiment, let the system stabilize for at least 3 days (tbc based on observations) and compare `library-data` and `library-data-expe` in terms of memory, CPU and Disk/IO ; also note any sensible change in terms of performance on live browsing by an end-user
- start with simple experiments:
- first, set all default values (just to confirm that our expected default values are correct and there is no bias)
- then, for every setting above, divide their value by 2 (one setting at a time, and all other settings are at their default value)
@rgaudin @mgautierfr @kelson42 WDYT?
Contributor guide
Research direction
Start with the library-data Kubernetes deployment and service, then review kiwix-serve's listed environment variables and their current defaults. Run the proposed staged experiments on library.kiwix.org, allowing stabilization between changes, and compare memory, CPU, disk/IO, and live browsing performance. Done means identifying safe cache settings and recording the observed results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100