PostHog / PostHog/posthog

ClickHouse Migration Error (DB::Exception: Table system.crash_log doesn't exist)

Open
#70,636 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug Description

This issue happens on fresh PostHog installation and when system.crash_log is absent on healthy ClickHouse instances
A fresh self-hosted PostHog installation can fail during ClickHouse migrations when using Docker.

There are two independent but related problems:

  1. ClickHouse Docker Version Compatibility (AVX2 Requirement):
    Recent ClickHouse Docker images require AVX2-capable CPUs, preventing deployment on older hardware.
    During testing, the first issue encountered was unrelated to PostHog itself, but to the default ClickHouse Docker image used by the self-hosted deployment.
    At the time of writing, the master branch of PostHog uses the following default ClickHouse image:
    clickhouse/clickhouse-server:26.6.1.1193
    This image requires a CPU with AVX2 instruction set support. AVX2 CPU flag, the ClickHouse server fails immediately during startup, preventing the PostHog stack from completing initialization.
    After testing several releases, the newest version that successfully started on my hardware was:
    clickhouse/clickhouse-server:26.5.5.8
    Changing only the Docker image inside docker-compose.base.yml version allowed the ClickHouse container to start normally

  2. Migration 0159 fails because system.crash_log is not created on a fresh, healthy ClickHouse instance

Migration posthog/clickhouse/migrations/0159_crash_log_metrics.py assumes system.crash_log already exists, but ClickHouse only creates this system table after a fatal server crash.
According to the ClickHouse documentation, system.crash_log is not created during installation or server startup.
Therefore, querying the table immediately after installation results in an exception similar to:

DB::Exception:
Table system.crash_log doesn't exist

Proposed Fix:
The attached migration applies the smallest possible change by disabling crash log metrics during the creation of the custom metrics view.
Original code :

CUSTOM_METRICS_VIEW(
    include_counters=True,
    include_server_crash=True,
)

Updated File (posthog/clickhouse/migrations/0159_crash_log_metrics.py):

from posthog.clickhouse.client.migration_tools import NodeRole, run_sql_with_exceptions
from posthog.clickhouse.custom_metrics import CUSTOM_METRICS_VIEW

operations = [
    run_sql_with_exceptions(
        CUSTOM_METRICS_VIEW(
            include_counters=True,
            include_server_crash=False,
        ),
        node_roles=[NodeRole.DATA],
    ),
]
Debug info

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with posthog/clickhouse/migrations/0159_crash_log_metrics.py and inspect how CUSTOM_METRICS_VIEW handles the absent system.crash_log table. Review docker-compose.base.yml for the ClickHouse image version and verify both fresh migration behavior and the reported CPU compatibility issue; done means a fresh self-hosted installation completes migrations without requiring system.crash_log.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, docker, python
Domain
databases, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.