cube-js / cube-js/cube

Template caches ignore schemaVersion, breaking documented behavior for dynamic data models

Open
#10,383 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 5h
Merged PRs (30d)
193

Description

## Problem

The [documentation for `schema_version`](https://cube.dev/docs/reference/configuration/config#schema_version) states:

> **schema_version can be used to tell Cube that the data model should be recompiled in case it depends on dynamic definitions fetched from some external database or API.**
>
> If the returned string is different, **the data model will be recompiled.**

However, when `schemaVersion` changes, the **template caches** (`compiledYamlCache`, `compiledJinjaCache`, `compiledScriptCache`) continue to serve stale output because they use only `MD5(file.content)` as the cache key—they don't include the schema version.

This means Jinja templates that "depend on dynamic definitions fetched from some external database or API" (the exact use case `schema_version` is documented for) return **cached output from a previous version** even after `schemaVersion` changes and recompilation is triggered.

## Minimal Reproduction

### Setup: Template with timestamp + schema_version that changes every request

```yaml
# model/cubes/test_recompile.yml.jinja
cubes:
- name: test_recompile
sql: "SELECT 1 as id, '{{ get_compile_timestamp() }}' as compiled_at"
dimensions:
- name: id
sql: id
type: number
primary_key: true
- name: compiled_at
sql: compiled_at
type: string
```

```python
# cube.py
from cube import config
from datetime import datetime
import random

@config("schema_version")
def schema_version(ctx: dict) -> str:
# Random value = every request triggers recompilation
return str(random.random())

def get_compile_timestamp() -> str:
# Returns current time when template is rendered
return datetime.now().isoformat()
```

### Expected behavior

Every request returns a different `schema_version`, so every request should recompile the schema. The `compiled_at` dimension should show a **new timestamp on every request**.

### Actual behavior

The `compiled_at` timestamp **never changes** after the first request. Despite `schema_version` changing and logs showing "Recompiling schema", the template function `get_compile_timestamp()` is never re-executed.

### What happens internally

1. Request arrives, `schema_version()` returns `"0.123"` (random)
2. `CompilerApi.getCompilers()` detects version differs from last request
3. Logs "Recompiling schema" and calls `DataSchemaCompiler.transpileJinjaFile()`
4. Cache key = `MD5(file.content)` → **cache hit** (file unchanged)
5. Returns cached template output from first-ever compilation
6. `get_compile_timestamp()` is **never called** - timestamp is stale forever

## Production Use Case

In real deployments, the template function would fetch dynamic configuration from a database or API:

```python
# Production example: template function fetches tenant-specific segments
def load_tenant_segments(tenant_id: int) -> list:
"""Called during template rendering to generate dynamic dimensions."""
response = requests.get(f"http://app/api/tenants/{tenant_id}/segments")
return response.json() # [{"name": "vip_users", "sql": "..."}, ...]
```

```yaml
# Production example: dynamically generate dimensions from API data
{% for segment in load_tenant_segments(COMPILE_CONTEXT.securityContext.tenant_id) %}
- name: {{ segment.name }}
sql: "{{ segment.sql }}"
type: string
{% endfor %}
```

When segment definitions change in the database, `schema_version` returns a new value to trigger recompilation—but the template cache prevents `load_tenant_segments()` from being called with fresh data.

## Root Cause

In `packages/cubejs-schema-compiler/src/compiler/DataSchemaCompiler.ts`, all three template caches use content-only keys:

```typescript
// Line 727 (YAML), 776 (Jinja), 854 (JS) - all identical pattern
const cacheKey = crypto.createHash('md5').update(JSON.stringify(file.content)).digest('hex');
```

The `schemaVersion` is not included in the cache key, so templates are served from cache even when the version changes and recompilation is triggered.

## Current Workaround (problematic)

The only workaround is to include schema-affecting config in `app_id`:

```python
@config("context_to_app_id")
def context_to_app_id(ctx: dict) -> str:
tenant_id = ctx["securityContext"]["tenant_id"]
tenant_config = get_tenant_config(tenant_id)
return f"APP_{tenant_id}_{tenant_config['segments_version']}"
```

This causes memory bloat and resource duplication because each `app_id` creates a new `CompilerApi` with its own resources. The [multitenancy docs explicitly warn about this overhead](https://cube.dev/docs/product/configuration/advanced/multitenancy#running-in-production).

## Environment

- Cube.js version: latest (2025-02)
- Using: Python SDK with Jinja templates
- Deployment: Multi-tenant SaaS with per-tenant dynamic configuration

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.