PostHog / PostHog/posthog

fix(warehouse-sources): a dropped source column auto-disables the schema instead of self-healing

Open
#98,692 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Problem

A dropped or renamed source column auto-disables a SQL warehouse schema, and the message points at the wrong cause.

  • All four SQL sources classify the driver's undefined-column error as non-retryable: Unknown column (mysql/source.py:302), does not exist (postgres/source.py:579, redshift/source.py:176), Invalid column name (mssql/source.py:111).
  • external_data_job.py:493 calls update_should_sync(..., should_sync=False, ...) on one occurrence, so syncing stops until a person re-enables the schema.
  • The MySQL message tells the customer to update the table's incremental field. That is not the cause when an ordinary data column was dropped.

Two paths reach it, and they differ in how long the failure lasts.

  • An explicit column selection is never reconciled against the live catalog at sync time. prune_enabled_columns runs only in reconcile_source_schema_metadata (common/sql/base.py:203), which the API view calls on source reload. A saved column name that no longer exists is therefore named on every run, so the schema stays disabled until someone reloads the source.
  • Sync-all resolves its projection from live discovery on the streaming connection (#95161). The same failure is possible only between that read and the query, and the next run would succeed.

The classification's own comment states the premise: the offending reference is persisted configuration, "almost always the configured incremental field", and "the streaming query reissues the same WHERE/ORDER BY on every attempt, so it fails identically forever" (mysql/source.py:292-301). That holds for the incremental field. It does not hold for a projection column.

Suggested fix

  • Catch the undefined-column error before the first batch, rediscover the catalog once, rebuild the projection and the Arrow schema, then retry the read. The "no batch yielded yet" guard already exists per source: yielded_any in mysql.py, yielded in _stream_arrow_batches in redshift.py, and the offset == 0 guards in postgres.py.
  • Prune an explicit selection against the rediscovered catalog on the same path, so a stale saved column stops being permanent.
  • Keep the non-retryable classification for the configured incremental field, where the premise still holds.

Context

Raised by reviewhog on #95161. That PR shrinks the sync-all window from minutes to a couple of round trips. It does not address the explicit-selection path, and it does not change the auto-disable response.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with external_data_job.py:493 and the undefined-column handling in mysql/source.py, postgres/source.py, redshift/source.py, and mssql/source.py. Then trace explicit-column reconciliation in common/sql/base.py:203 and the pre-first-batch guards named in mysql.py, redshift.py, and postgres.py. Done means stale projection columns can self-heal without permanently disabling the schema, while configured incremental-field failures retain their non-retryable behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, postgres, postgresql, python, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.