apache / apache/texera

local-dev.sh: schema-currency check only probes the `feedback` table, silently skips DDL on a stale DB → jOOQ codegen fails with a misleading compile error

Open
#6,194 2 comments 0 reactions 0 assignees View on GitHub
infra
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

`bin/local-dev.sh up` decides whether the Postgres schema needs (re)loading by probing for a **single hard-coded table, `feedback`** (`bin/local-dev/main.sh`, `infra_ensure_db_schema`, ~L1346-1357):

```bash
# `feedback` is one of the newer tables; if it exists we assume the
# whole schema is current.
has_feedback=$(docker exec "$pg" psql -U texera -d texera_db -tAc \
"SELECT 1 FROM pg_tables WHERE schemaname='texera_db' AND tablename='feedback'" ...)
if [[ "$has_feedback" == "1" ]]; then
tui_skip "postgres: schema already current"
return 0
fi
```

The probe assumes `feedback` is the newest table. Any table added *after* `feedback` is invisible to this check. When the Postgres data volume was created before such a table existed (a very common case for a long-lived local dev volume), the volume already has `feedback`, so the script reports **"postgres: schema already current"** and skips applying `sql/texera_ddl.sql` — even though the schema is actually stale.

The downstream effect is nasty: the `dao` module runs jOOQ codegen against this stale live DB (`common/dao/build.sbt`, wired into `Compile / sourceGenerators`), so the generated `Tables`/`Keys` are missing the new relation, and `sbt dist` then fails with a **misleading compile error** that points at application Scala code, not at the schema. In my case the missing table was `workflow_cover_image` (added in `sql/updates/27.sql`), and the build failed with 11 errors like:

```
not found: value WORKFLOW_COVER_IMAGE
```

Expected: the schema-currency check should not silently skip DDL when the schema is out of date. Since `sql/texera_ddl.sql` is already idempotent (`CREATE TABLE IF NOT EXISTS`), the safest fix is to **always apply it** (the comment in the code even acknowledges re-applying is safe), or to probe the newest expected table / a schema-version marker instead of `feedback`.

### How to reproduce?

1. Have a `texera-local-dev` Postgres volume created before `workflow_cover_image` was added (i.e. an older volume that already contains `feedback`).
2. Check out a branch whose code references the newer table (references `WORKFLOW_COVER_IMAGE`).
3. Run `bin/local-dev.sh up`.
4. Observe `postgres: schema already current` is printed (DDL skipped), then `sbt: dist FAILED` with `not found: value WORKFLOW_COVER_IMAGE` — an error that gives no hint that the real cause is a stale DB schema.

Workaround: manually apply the missing migration (e.g. `docker exec -i texera-postgres psql -U texera -d texera_db < sql/updates/27.sql`) and re-run `up`.

### Version/Branch

1.3.0-incubating-SNAPSHOT (main)

### Commit Hash (Optional)

be99b3661 (observed; the check lives in `bin/local-dev/main.sh`)

### Relevant log output

```shell
▸ Infra
✓ infra: 5 containers up
○ postgres: schema already current

▸ Build
→ sbt dist (log: /tmp/texera-local-dev/logs/sbt-dist.log) (no-TTY)
✗ sbt: dist FAILED

# sbt-dist.log:
[error] .../WorkflowResource.scala:729:15: not found: value WORKFLOW_COVER_IMAGE
[error] .../WorkflowResource.scala:730:13: not found: value WORKFLOW_COVER_IMAGE
...
[error] 11 errors found
[error] (WorkflowExecutionService / Compile / compileIncremental) Compilation failed
```

Contributor guide

Open the contributing guide

Research direction

Start in bin/local-dev/main.sh around infra_ensure_db_schema and inspect how bin/local-dev.sh up decides whether to apply sql/texera_ddl.sql. Check the dao codegen wiring in common/dao/build.sbt and reproduce with a stale Postgres volume containing feedback but missing workflow_cover_image. Done means schema currency is detected reliably and sbt dist no longer fails because generated jOOQ relations are missing.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, postgresql, scala, shell
Domain
build-system, database, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.