Automattic / Automattic/jetpack
Sitemaps: image sitemap build accumulates memory linearly, OOMs cron on 100k+ media
- Dominant language
- PHP
- Stars
- 1.8k
- Forks
- 898
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 774
Description
### Impacted plugin
Jetpack
### Quick summary
On large media libraries, Jetpack's **image sitemap** build (`jp_sitemap_cron_hook` → `jp_img_sitemap` / `image-sitemap-N.xml`) grows resident memory ~**10 MB per chunk** and never frees it within the request, so it walks up to / past the PHP `memory_limit` and OOMs during WP-Cron. Regular sync crons stay flat — the cost is entirely the sitemap builder.
### Steps to reproduce
1. Site with a large media library (here: **over 113,000** media items, **over 16,000** posts, several CPTs, WooCommerce + Subscriptions) so the image sitemap spans 100+ chunks.
2. Default PHP `memory_limit` (512M).
3. Let `jp_sitemap_cron_hook` run (or trigger it).
4. Watch memory climb ~10 MB per `image-sitemap-N.xml` chunk (~390 KB each) and approach/exceed 512M → OOM fatal (surfaces in `wp-includes/class-wpdb.php`, the victim of the next allocation).
### Site owner impact
Fewer than 20% of the total website/platform users
### Severity
Minor
### What other impact(s) does this issue have?
No revenue impact
### If a workaround is available, please outline it here.
Per-chunk climb in one run:
```
cron START: jp_sitemap_cron_hook 166 MB
image-sitemap-96.xml (397 KB) 176 MB
image-sitemap-98.xml 196 MB
image-sitemap-100.xml 214 MB
image-sitemap-103.xml 244 MB
… → 404 MB
```
Peak per sitemap-cron run (UTC):
| Run | Peak |
|---|---|
| Jun 11 (from 12:22) | did not run |
| Jun 12 21:07 | 412 MB |
| Jun 13 09:06 | 513 MB (over limit) |
| Jun 13 21:07 | 511 MB |
| Jun 14 21:06 | 434 MB |
| Added mitigation | |
| Jun 15 | did not run|
| Jun 16 09:07 | 404 MB |
| Jun 17 | did not run |
| Jun 18 09:06 | 312 MB |
| Jun 19 (to 09:14) | did not run |
- Jun 13 21:07 hit 511 MB writing **zero** new chunks — the build holds the memory regardless of writes.
- **Mitigation tried:** `add_filter( 'jetpack_sitemap_suspend_cache_addition', '__return_true' )` (the 15.0 filter). Stopped the hard OOMs, but the run still climbs to ~300–400 MB — so it's not just the object cache; the builder/data itself accumulates.
- **Suggested fix:** stagger and resume — cap work per run by item count (filterable, e.g. ~75,000 media items), persist state, continue next cron run so memory is reclaimed between runs; and/or free per-chunk data (`DOMDocument`/buffer, primed caches) after each chunk.
### Platform (Simple and/or Atomic)
Atomic
Contributor guide
Research direction
Start by tracing the jp_sitemap_cron_hook flow into jp_sitemap_cron_hook and jp_img_sitemap, then reproduce the memory growth with a large media library. Inspect how each image-sitemap-N.xml chunk retains DOMDocument, buffers, and cache data. Done means the image sitemap build no longer accumulates memory until the request exceeds its limit, with coverage for multi-chunk runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php, wordpress
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100