Automattic / Automattic/jetpack

Sitemaps: image sitemap build accumulates memory linearly, OOMs cron on 100k+ media

Open
#49,786 1 comment 0 reactions 0 assignees View on GitHub
Bug Needs triage
Dominant language
PHP
Stars
1.8k
Forks
898
Avg merge
1d 18h
Merged PRs (30d)
774

Description

### Impacted plugin

Jetpack

### Quick summary

On large media libraries, Jetpack's **image sitemap** build (`jp_sitemap_cron_hook` → `jp_img_sitemap` / `image-sitemap-N.xml`) grows resident memory ~**10 MB per chunk** and never frees it within the request, so it walks up to / past the PHP `memory_limit` and OOMs during WP-Cron. Regular sync crons stay flat — the cost is entirely the sitemap builder.

### Steps to reproduce

1. Site with a large media library (here: **over 113,000** media items, **over 16,000** posts, several CPTs, WooCommerce + Subscriptions) so the image sitemap spans 100+ chunks.
2. Default PHP `memory_limit` (512M).
3. Let `jp_sitemap_cron_hook` run (or trigger it).
4. Watch memory climb ~10 MB per `image-sitemap-N.xml` chunk (~390 KB each) and approach/exceed 512M → OOM fatal (surfaces in `wp-includes/class-wpdb.php`, the victim of the next allocation).

### Site owner impact

Fewer than 20% of the total website/platform users

### Severity

Minor

### What other impact(s) does this issue have?

No revenue impact

### If a workaround is available, please outline it here.

Per-chunk climb in one run:
```
cron START: jp_sitemap_cron_hook 166 MB
image-sitemap-96.xml (397 KB) 176 MB
image-sitemap-98.xml 196 MB
image-sitemap-100.xml 214 MB
image-sitemap-103.xml 244 MB
… → 404 MB
```

Peak per sitemap-cron run (UTC):

| Run | Peak |
|---|---|
| Jun 11 (from 12:22) | did not run |
| Jun 12 21:07 | 412 MB |
| Jun 13 09:06 | 513 MB (over limit) |
| Jun 13 21:07 | 511 MB |
| Jun 14 21:06 | 434 MB |
| Added mitigation | |
| Jun 15 | did not run|
| Jun 16 09:07 | 404 MB |
| Jun 17 | did not run |
| Jun 18 09:06 | 312 MB |
| Jun 19 (to 09:14) | did not run |

- Jun 13 21:07 hit 511 MB writing **zero** new chunks — the build holds the memory regardless of writes.
- **Mitigation tried:** `add_filter( 'jetpack_sitemap_suspend_cache_addition', '__return_true' )` (the 15.0 filter). Stopped the hard OOMs, but the run still climbs to ~300–400 MB — so it's not just the object cache; the builder/data itself accumulates.
- **Suggested fix:** stagger and resume — cap work per run by item count (filterable, e.g. ~75,000 media items), persist state, continue next cron run so memory is reclaimed between runs; and/or free per-chunk data (`DOMDocument`/buffer, primed caches) after each chunk.

### Platform (Simple and/or Atomic)

Atomic

Contributor guide

Open the contributing guide

Research direction

Start by tracing the jp_sitemap_cron_hook flow into jp_sitemap_cron_hook and jp_img_sitemap, then reproduce the memory growth with a large media library. Inspect how each image-sitemap-N.xml chunk retains DOMDocument, buffers, and cache data. Done means the image sitemap build no longer accumulates memory until the request exceeds its limit, with coverage for multi-chunk runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, wordpress
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.