cube-js / cube-js/cube

Refresh worker heap grows until OOM when a partitioned pre-aggregation's build range moves with time

Open
#11,860 1 comment 0 reactions 1 assignee Claimed by @ovr View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 10h
Merged PRs (30d)
203

Description

**Describe the bug**

When a partitioned pre-aggregation has a build range that changes over time — e.g. `buildRangeStart: { sql: 'SELECT NOW() - INTERVAL 3 YEAR' }` — the refresh worker's heap grows linearly until it reaches the V8 heap limit and the process dies. It restarts, and the cycle repeats.

How it happens:

1. The refresh scheduler expands partitions through `compilerApi.compilerCacheFn(requestId, baseQuery, ['expandPartitions'])` ([RefreshScheduler.ts#L170](https://github.com/cube-js/cube/blob/8368f2af49/packages/cubejs-server-core/src/core/RefreshScheduler.ts#L170)). `baseQuery` is the same for a given pre-aggregation and timezone on every run.
2. `PreAggregationPartitionRangeLoader.partitionPreAggregations()` caches the whole list of partition descriptions under `['partitions', JSON.stringify(buildRange)]` ([PreAggregationPartitionRangeLoader.ts#L416](https://github.com/cube-js/cube/blob/8368f2af49/packages/cubejs-query-orchestrator/src/orchestrator/PreAggregationPartitionRangeLoader.ts#L416)).
3. That cache is a schema-compiler `QueryCache`: a plain nested object with no size limit and no eviction ([QueryCache.ts#L5](https://github.com/cube-js/cube/blob/8368f2af49/packages/cubejs-schema-compiler/src/adapter/QueryCache.ts#L5), [#L22](https://github.com/cube-js/cube/blob/8368f2af49/packages/cubejs-schema-compiler/src/adapter/QueryCache.ts#L22)). It lives in the `CompilerCache` LRU, which uses `updateAgeOnGet: true` ([CompilerCache.ts#L15](https://github.com/cube-js/cube/blob/8368f2af49/packages/cubejs-schema-compiler/src/compiler/CompilerCache.ts#L15)). The scheduler reads the entry on every run, so it never expires.
4. The build range query result is renewed together with the refresh key. If it depends on the current time, every renewal yields a new `buildRange`, a new list is stored, and none of the previous ones is ever released.

Each description carries its own copy of the partition's SQL (`loadSql`, `sql`, `structureVersionLoadSql`, …). A single list for a day-partitioned rollup over a 3-year window (~1,100 partitions) therefore weighs several MB.

**Impact**

In production, with about ten day-partitioned rollups, two `scheduledRefreshTimeZones`, a 3-year window and 15-minute refresh keys, the refresh worker's heap grew by ~150 MB/h. With `--max-old-space-size=1536` it died every ~11.5 hours, for two months, in two environments. Pre-aggregation builds in flight at the time of the crash are lost and have to be picked up again.

A sampling heap profile of the live process attributed 73 % of the memory allocated during a 10-minute window, and still live at its end, to `partitionPreAggregationDescription` called from the cached closure in `partitionPreAggregations()`.

**To Reproduce**

A standalone script against the published packages. It needs no database: it stubs `loadBuildRange` and drives the loader the way the refresh scheduler does, once per refresh-key renewal.

```
npm i @cubejs-backend/query-orchestrator@1.7.37 @cubejs-backend/schema-compiler@1.7.37 @cubejs-backend/shared@1.7.37
node --expose-gc repro.js moving
node --expose-gc repro.js stable
```

repro.js

```javascript
// node --expose-gc repro.js [renewals]
// Simulates the refresh scheduler expanding day-partitioned rollups, once per refresh-key renewal.
const { PreAggregationPartitionRangeLoader } = require('@cubejs-backend/query-orchestrator/dist/src/orchestrator/PreAggregationPartitionRangeLoader');
const { CompilerCache } = require('@cubejs-backend/schema-compiler/dist/src/compiler/CompilerCache');
const { FROM_PARTITION_RANGE, TO_PARTITION_RANGE } = require('@cubejs-backend/shared');

const mode = process.argv[2] || 'moving';
const renewals = Number(process.argv[3] || 12);
const ROLLUPS = 5;
const TIMEZONES = ['UTC', 'Europe/Paris'];

// Same defaults as the server: max 10000, ttl 10 min, updateAgeOnGet: true
const compilerCache = new CompilerCache({});

function preAggregation(i, timezone) {
const tableName = `pre_aggregations.rollup${i}`;
const params = [FROM_PARTITION_RANGE, TO_PARTITION_RANGE];
const sql = `SELECT ${'col, '.repeat(800)}x FROM t WHERE ts >= ? AND ts <= ?`; // a realistic ~4 KB rollup query
return {
preAggregationId: `Cube${i}.rollup`, tableName, timezone, dataSource: 'default', type: 'rollup',
partitionGranularity: 'day', timestampFormat: 'YYYY-MM-DDTHH:mm:ss.SSS', timestampPrecision: 3,
loadSql: [`CREATE TABLE ${tableName} AS ${sql}`, params, {}],
sql: [sql, params, {}],
invalidateKeyQueries: [['SELECT FLOOR(UNIX_TIMESTAMP() / 900)', [], { renewalThreshold: 90 }]],
};
}

// buildRangeStart: `SELECT NOW() - INTERVAL 3 YEAR` (moving) vs first day of the month (stable).
// The build range result is renewed with the refresh key (every 15 min here).
function buildRange(renewal) {
const now = new Date(Date.UTC(2026, 8, 12) + renewal * 15 * 60 * 1000 + 7000);
const start = new Date(now);
start.setUTCFullYear(now.getUTCFullYear() - 3);
const iso = start.toISOString();
const s = mode === 'stable' ? `${iso.slice(0, 8)}01T00:00:00.000` : `${iso.slice(0, 19)}.000`;
return [s, '2026-10-01T00:00:00.000'];
}

async function renew(r) {
for (let i = 0; i < ROLLUPS; i++) {
for (const timezone of TIMEZONES) {
const pa = preAggregation(i, timezone);
// RefreshScheduler: compilerCacheFn(requestId, baseQuery, ['expandPartitions']) — baseQuery is stable per rollup
const baseQuery = { timezone, preAggregationId: pa.preAggregationId };
const compilerCacheFn = (subKey, fn) => compilerCache.getQueryCache(baseQuery).cache(['expandPartitions'].concat(subKey), fn);
const loader = new PreAggregationPartitionRangeLoader(null, () => {}, null, null, pa, {}, null, { maxPartitions: 10000, compilerCacheFn });
loader.loadBuildRange = async () => buildRange(r);
await loader.partitionPreAggregations();
}
}
}

(async () => {
const heapMb = () => { global.gc(); return process.memoryUsage().heapUsed / 1048576; };
const base = heapMb();
for (let r = 1; r <= renewals; r++) {
await renew(r);
if (r % 4 === 0) {
let lists = 0;
for (const [, qc] of compilerCache.queryCache.entries()) lists += Object.keys(qc.storage.expandPartitions.partitions).length;
console.log(`${mode}: ${r / 4} h of renewals -> heap +${(heapMb() - base).toFixed(0)} MB, cached partition lists: ${lists}`);
}
}
})();
```

Output, identical on 1.6.20 and 1.7.37:

```
moving: 1 h of renewals -> heap +84 MB, cached partition lists: 40
moving: 2 h of renewals -> heap +167 MB, cached partition lists: 80
moving: 3 h of renewals -> heap +250 MB, cached partition lists: 120
stable: 1 h of renewals -> heap +21 MB, cached partition lists: 10
stable: 2 h of renewals -> heap +21 MB, cached partition lists: 10
stable: 3 h of renewals -> heap +21 MB, cached partition lists: 10
```

**Expected behavior**

The memory held for partition descriptions stays bounded. Either lists for a previous build range are released once the range moves, or the cache key does not depend on a value that changes on every renewal.

**Minimally reproducible Cube Schema**

Any partitioned rollup whose build range moves with time triggers it (MySQL syntax):

```javascript
cube(`Events`, {
sql: `SELECT 1 AS id, NOW() AS created_at`,
measures: {
count: { type: `count` },
},
dimensions: {
createdAt: { sql: `created_at`, type: `time` },
},
preAggregations: {
byDay: {
type: `rollup`,
measures: [count],
timeDimension: createdAt,
granularity: `day`,
partitionGranularity: `day`,
refreshKey: { every: `15 minute`, incremental: true, updateWindow: `7 day` },
buildRangeStart: { sql: `SELECT NOW() - INTERVAL 3 YEAR` },
buildRangeEnd: { sql: `SELECT NOW()` },
},
},
});
```

**Version:**

Reproduced on 1.6.20 and 1.7.37. The code involved is unchanged on `master` as of 8368f2a.

**Additional context**

Workaround: make both bounds change rarely, for example by aligning them on the month (MySQL):

```sql
-- buildRangeStart: first day of the current month, 3 years back
SELECT DATE_SUB(DATE_SUB(CURDATE(), INTERVAL DAYOFMONTH(CURDATE()) - 1 DAY), INTERVAL 3 YEAR)
-- buildRangeEnd: first day of the next month
SELECT ADDDATE(LAST_DAY(NOW()), 1)
```

The key then changes once a month and the heap stays flat. The window becomes slightly wider than intended, and one extra list still accumulates each month until the process restarts. We have not tested the default `MIN`/`MAX` build range, but it would change whenever new rows arrive, with the same effect.

#10247 addressed exactly this: it replaced the nested-object `QueryCache` with a `Map` and put an LRU limit on `CompilerCache`. It was approved, then closed without merging on 2026-04-15. Reviving it would fix the leak. A narrower alternative would be to stop keying the cached list by the whole build range: for example, cache each partition's description by its own (stable) range, or drop the previous `partitions` entries of a `baseQuery` when a new build range is stored.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.