PostHog / PostHog/posthog

Hobby deployment failing - Clickhouse errors

Open
#25,533 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
7h 27m
Merged PRs (30d)
222

Description

Bug Description

When attempting to run the hobby deployment script after 10 minutes the deployment does not update.
Checking the docker containers with ps I see four containers continuously restarting.

  1. clickhouse
  2. plugins
  3. web
  4. worker

I believe I've deduced the issue to the click house not starting due to a dependency issue.

How to reproduce

  1. Run hobby deployment script from docs
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)"
  2. Wait for the 5-10 minutes
  3. Check docker logs for click house

Additional context

I have run the upgrade script a few times over the last few days after viewing some other similar click house issues being resolved.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/upgrade-hobby)"

I am running this on a GCP VM. The image was built with the following.

  • Canonical, Ubuntu, 20.04 LTS, amd64 focal image built on 2024-08-30

I've pasted the error logs from the click house container below.

Debug info
/entrypoint.sh: running /docker-entrypoint-initdb.d/init-db.sh
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
python3.9 is already the newest version (3.9.5-3ubuntu0~20.04.1).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libc-bin : Depends: libc6 (< 2.32) but 2.35-0ubuntu3 is to be installed
 libc6 : Breaks: locales (< 2.35) but 2.31-0ubuntu9.14 is to be installed
         Recommends: libnss-nis but it is not installable
         Recommends: libnss-nisplus but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with bin/deploy-hobby and inspect the ClickHouse container logs, especially /entrypoint.sh and /docker-entrypoint-initdb.d/init-db.sh. Compare the behavior after bin/upgrade-hobby and trace the apt dependency errors. Done means the deployment completes without the ClickHouse, plugins, web, and worker containers continuously restarting.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, docker, gcp, shell, ubuntu
Domain
cloud, databases, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.