kiwix / kiwix/operations

Customize kiwix-serve cache settings to limit memory consumption

Open
#147 8 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
15
Forks
5
PR merge metrics
No merged PRs in 30d

Description

As of today, kiwix-serve cache settings are not customized on library.kiwix.org (and not on dev.library.kiwix.org)

As discussed in https://github.com/kiwix/libkiwix/issues/1025, kiwix-serve is using a significant amount of memory. With current code, we could probably put more control on this memory consumption by customizing some settings explained below

| Environment variable | Purpose | Default value | Comment |
|---|---|---|---|
| `KIWIX_ARCHIVE_CACHE_SIZE` | Number of open readers (~ZIM) | 10% of `getBookCount_not_protected` (number of local and remote books) ~= 421 today | |
| `KIWIX_SEARCHER_CACHE_SIZE` | Number of open searcher (which might include readers non accounted for in `KIWIX_ARCHIVE_CACHE_SIZE`) | idem `KIWIX_ARCHIVE_CACHE_SIZE` ~= 421 today | |
| `ZIM_DIRENTCACHE` | Number of dirent kept in cache per ZIM | 512 | Probably low impact on memory |
| `ZIM_DIRENTLOOKUPCACHE` | Idem `ZIM_DIRENTCACHE` | 1024 | Probably low impact on memory |
| `ZIM_CLUSTERCACHE` | Number of cluster kept in cache per ZIM | 16 | |

My gut feeling is that 412 for `KIWIX_ARCHIVE_CACHE_SIZE` and `KIWIX_SEARCHER_CACHE_SIZE` is way too much, I wouldn't assume we open this amount of ZIM every day, but my experience is limited.

I suggest that we do a small experiments directly in production on library.kiwix.org (dev.library.kiwix.org is not really pertinent in terms of number of ZIMs + traffic and has known issues):
- instead of having 2 kiwix-serve containers in deployment `library-data`, reduce this to 1
- create a new deployment `library-data-expe`, with 1 kiwix-serve container and custom environment variables
- modify `library-data` service to redirect to both k8s deployment
- this is easy to do (easier than at varnish side)
- should we encounter a problem, we just scale `library-data` to 2 containers and `library-data-expe` to 0 et voilà
- for every experiment, let the system stabilize for at least 3 days (tbc based on observations) and compare `library-data` and `library-data-expe` in terms of memory, CPU and Disk/IO ; also note any sensible change in terms of performance on live browsing by an end-user
- start with simple experiments:
- first, set all default values (just to confirm that our expected default values are correct and there is no bias)
- then, for every setting above, divide their value by 2 (one setting at a time, and all other settings are at their default value)

@rgaudin @mgautierfr @kelson42 WDYT?

Contributor guide

Open the contributing guide

Research direction

Start with the library-data Kubernetes deployment and service, then review kiwix-serve's listed environment variables and their current defaults. Run the proposed staged experiments on library.kiwix.org, allowing stabilization between changes, and compare memory, CPU, disk/IO, and live browsing performance. Done means identifying safe cache settings and recording the observed results.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.