Template caches ignore schemaVersion, breaking documented behavior for dynamic data models
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 193
Description
## Problem
The [documentation for `schema_version`](https://cube.dev/docs/reference/configuration/config#schema_version) states:
> **schema_version can be used to tell Cube that the data model should be recompiled in case it depends on dynamic definitions fetched from some external database or API.**
>
> If the returned string is different, **the data model will be recompiled.**
However, when `schemaVersion` changes, the **template caches** (`compiledYamlCache`, `compiledJinjaCache`, `compiledScriptCache`) continue to serve stale output because they use only `MD5(file.content)` as the cache key—they don't include the schema version.
This means Jinja templates that "depend on dynamic definitions fetched from some external database or API" (the exact use case `schema_version` is documented for) return **cached output from a previous version** even after `schemaVersion` changes and recompilation is triggered.
## Minimal Reproduction
### Setup: Template with timestamp + schema_version that changes every request
```yaml
# model/cubes/test_recompile.yml.jinja
cubes:
- name: test_recompile
sql: "SELECT 1 as id, '{{ get_compile_timestamp() }}' as compiled_at"
dimensions:
- name: id
sql: id
type: number
primary_key: true
- name: compiled_at
sql: compiled_at
type: string
```
```python
# cube.py
from cube import config
from datetime import datetime
import random
@config("schema_version")
def schema_version(ctx: dict) -> str:
# Random value = every request triggers recompilation
return str(random.random())
def get_compile_timestamp() -> str:
# Returns current time when template is rendered
return datetime.now().isoformat()
```
### Expected behavior
Every request returns a different `schema_version`, so every request should recompile the schema. The `compiled_at` dimension should show a **new timestamp on every request**.
### Actual behavior
The `compiled_at` timestamp **never changes** after the first request. Despite `schema_version` changing and logs showing "Recompiling schema", the template function `get_compile_timestamp()` is never re-executed.
### What happens internally
1. Request arrives, `schema_version()` returns `"0.123"` (random)
2. `CompilerApi.getCompilers()` detects version differs from last request
3. Logs "Recompiling schema" and calls `DataSchemaCompiler.transpileJinjaFile()`
4. Cache key = `MD5(file.content)` → **cache hit** (file unchanged)
5. Returns cached template output from first-ever compilation
6. `get_compile_timestamp()` is **never called** - timestamp is stale forever
## Production Use Case
In real deployments, the template function would fetch dynamic configuration from a database or API:
```python
# Production example: template function fetches tenant-specific segments
def load_tenant_segments(tenant_id: int) -> list:
"""Called during template rendering to generate dynamic dimensions."""
response = requests.get(f"http://app/api/tenants/{tenant_id}/segments")
return response.json() # [{"name": "vip_users", "sql": "..."}, ...]
```
```yaml
# Production example: dynamically generate dimensions from API data
{% for segment in load_tenant_segments(COMPILE_CONTEXT.securityContext.tenant_id) %}
- name: {{ segment.name }}
sql: "{{ segment.sql }}"
type: string
{% endfor %}
```
When segment definitions change in the database, `schema_version` returns a new value to trigger recompilation—but the template cache prevents `load_tenant_segments()` from being called with fresh data.
## Root Cause
In `packages/cubejs-schema-compiler/src/compiler/DataSchemaCompiler.ts`, all three template caches use content-only keys:
```typescript
// Line 727 (YAML), 776 (Jinja), 854 (JS) - all identical pattern
const cacheKey = crypto.createHash('md5').update(JSON.stringify(file.content)).digest('hex');
```
The `schemaVersion` is not included in the cache key, so templates are served from cache even when the version changes and recompilation is triggered.
## Current Workaround (problematic)
The only workaround is to include schema-affecting config in `app_id`:
```python
@config("context_to_app_id")
def context_to_app_id(ctx: dict) -> str:
tenant_id = ctx["securityContext"]["tenant_id"]
tenant_config = get_tenant_config(tenant_id)
return f"APP_{tenant_id}_{tenant_config['segments_version']}"
```
This causes memory bloat and resource duplication because each `app_id` creates a new `CompilerApi` with its own resources. The [multitenancy docs explicitly warn about this overhead](https://cube.dev/docs/product/configuration/advanced/multitenancy#running-in-production).
## Environment
- Cube.js version: latest (2025-02)
- Using: Python SDK with Jinja templates
- Deployment: Multi-tenant SaaS with per-tenant dynamic configuration
Contributor guide
Assessment
This issue has not been assessed yet.