ClickHouse Migration Error (DB::Exception: Table system.crash_log doesn't exist)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug Description
This issue happens on fresh PostHog installation and when system.crash_log is absent on healthy ClickHouse instances
A fresh self-hosted PostHog installation can fail during ClickHouse migrations when using Docker.
There are two independent but related problems:
-
ClickHouse Docker Version Compatibility (AVX2 Requirement):
Recent ClickHouse Docker images require AVX2-capable CPUs, preventing deployment on older hardware.
During testing, the first issue encountered was unrelated to PostHog itself, but to the default ClickHouse Docker image used by the self-hosted deployment.
At the time of writing, the master branch of PostHog uses the following default ClickHouse image:
clickhouse/clickhouse-server:26.6.1.1193
This image requires a CPU with AVX2 instruction set support. AVX2 CPU flag, the ClickHouse server fails immediately during startup, preventing the PostHog stack from completing initialization.
After testing several releases, the newest version that successfully started on my hardware was:
clickhouse/clickhouse-server:26.5.5.8
Changing only the Docker image insidedocker-compose.base.ymlversion allowed the ClickHouse container to start normally -
Migration 0159 fails because system.crash_log is not created on a fresh, healthy ClickHouse instance
Migration posthog/clickhouse/migrations/0159_crash_log_metrics.py assumes system.crash_log already exists, but ClickHouse only creates this system table after a fatal server crash.
According to the ClickHouse documentation, system.crash_log is not created during installation or server startup.
Therefore, querying the table immediately after installation results in an exception similar to:
DB::Exception:
Table system.crash_log doesn't exist
Proposed Fix:
The attached migration applies the smallest possible change by disabling crash log metrics during the creation of the custom metrics view.
Original code :
CUSTOM_METRICS_VIEW(
include_counters=True,
include_server_crash=True,
)
Updated File (posthog/clickhouse/migrations/0159_crash_log_metrics.py):
from posthog.clickhouse.client.migration_tools import NodeRole, run_sql_with_exceptions
from posthog.clickhouse.custom_metrics import CUSTOM_METRICS_VIEW
operations = [
run_sql_with_exceptions(
CUSTOM_METRICS_VIEW(
include_counters=True,
include_server_crash=False,
),
node_roles=[NodeRole.DATA],
),
]
Debug info
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with posthog/clickhouse/migrations/0159_crash_log_metrics.py and inspect how CUSTOM_METRICS_VIEW handles the absent system.crash_log table. Review docker-compose.base.yml for the ClickHouse image version and verify both fresh migration behavior and the reported CPU compatibility issue; done means a fresh self-hosted installation completes migrations without requiring system.crash_log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, docker, python
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100