fix(warehouse-sources): a dropped source column auto-disables the schema instead of self-healing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Problem
A dropped or renamed source column auto-disables a SQL warehouse schema, and the message points at the wrong cause.
- All four SQL sources classify the driver's undefined-column error as non-retryable:
Unknown column(mysql/source.py:302),does not exist(postgres/source.py:579,redshift/source.py:176),Invalid column name(mssql/source.py:111). external_data_job.py:493callsupdate_should_sync(..., should_sync=False, ...)on one occurrence, so syncing stops until a person re-enables the schema.- The MySQL message tells the customer to update the table's incremental field. That is not the cause when an ordinary data column was dropped.
Two paths reach it, and they differ in how long the failure lasts.
- An explicit column selection is never reconciled against the live catalog at sync time.
prune_enabled_columnsruns only inreconcile_source_schema_metadata(common/sql/base.py:203), which the API view calls on source reload. A saved column name that no longer exists is therefore named on every run, so the schema stays disabled until someone reloads the source. - Sync-all resolves its projection from live discovery on the streaming connection (#95161). The same failure is possible only between that read and the query, and the next run would succeed.
The classification's own comment states the premise: the offending reference is persisted configuration, "almost always the configured incremental field", and "the streaming query reissues the same WHERE/ORDER BY on every attempt, so it fails identically forever" (mysql/source.py:292-301). That holds for the incremental field. It does not hold for a projection column.
Suggested fix
- Catch the undefined-column error before the first batch, rediscover the catalog once, rebuild the projection and the Arrow schema, then retry the read. The "no batch yielded yet" guard already exists per source:
yielded_anyinmysql.py,yieldedin_stream_arrow_batchesinredshift.py, and theoffset == 0guards inpostgres.py. - Prune an explicit selection against the rediscovered catalog on the same path, so a stale saved column stops being permanent.
- Keep the non-retryable classification for the configured incremental field, where the premise still holds.
Context
Raised by reviewhog on #95161. That PR shrinks the sync-all window from minutes to a couple of round trips. It does not address the explicit-selection path, and it does not change the auto-disable response.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with external_data_job.py:493 and the undefined-column handling in mysql/source.py, postgres/source.py, redshift/source.py, and mssql/source.py. Then trace explicit-column reconciliation in common/sql/base.py:203 and the pre-first-batch guards named in mysql.py, redshift.py, and postgres.py. Done means stale projection columns can self-heal without permanently disabling the schema, while configured incremental-field failures retain their non-retryable behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, postgres, postgresql, python, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100