PostHog / PostHog/chschema

feat(locate): report semantic column references for safe drops

Open
#236 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
15
Forks
2
Avg merge
1h 1m
Merged PRs (30d)
27

Description

Problem

After #234 and #235, table/column lookup can answer which resolved models contain a selected column. It cannot answer which modeled objects and expressions reference that column.

A user preparing to drop a known resolved column needs a reverse-reference report across every manifest composition, layer, and node dump. Text search is insufficient: it produces unrelated matches, misses semantic references through qualification and aliases, and cannot account for inheritance, patches, overrides, or drops.

The current validator dependency graph is primarily object-level. Diff and plan can emit a DROP COLUMN operation without proving that the column is unused.

Proposed interface

Extend locate with a reference mode:

hclexp locate \
  -manifest manifest.hcl \
  -layer-root ./schema \
  -tables posthog.events \
  -columns person_properties \
  -references \
  -format json

Both selectors remain required. Reference mode should report reverse references to every selected resolved column rather than only reporting model membership.

Model-loading semantics

Reference lookup must retain the model-first behavior:

  1. A manifest produces one resolved model per role/environment stack.
  2. The comma-separated -layer list is loaded in declared order as one resolved model.
  3. Every HCL file under -dump is loaded as an independent node model.
  4. Independent models load through the existing bounded parallel worker pool. A 30-node cluster must be loaded concurrently, while output remains deterministic.
  5. References are analyzed only after composition and resolution, so inherited, patched, modified, overridden, and dropped state is respected.

References to cover

Within a table:

  • primary_key and order_by
  • partition_by, sample_by, and table TTL
  • column DEFAULT, MATERIALIZED, EPHEMERAL, ALIAS, and column TTL expressions
  • indexes
  • projections
  • CHECK and ASSUME constraints
  • engine arguments that name columns

Across objects:

  • materialized-view and view queries, including qualified names and aliases
  • SELECT * and table.* wildcard projections
  • materialized-view destination shape
  • Distributed proxy/remote column contracts
  • Buffer and TimeSeries forwarding/target contracts where applicable
  • dictionary source queries and attribute expressions where they can reference the selected table column

The analysis must resolve SQL scopes rather than matching identifier text. The same column name on an unrelated table is not a reference.

Provenance and output

Each result should identify:

  • target database, table, and column
  • model source: manifest role/environment, ordered layer stack, or dump file/node
  • referring object type and qualified name
  • reference kind, such as order_by, projection, view_query, or distributed_contract
  • relevant expression or query
  • authored file, line, and layer when source provenance is available

References shared by several compositions should be deduplicated as source sites while retaining every affected model placement.

JSON output should be stable and machine-readable. It should distinguish definite references from unresolved safety blockers.

Fail-closed behavior

The command must not claim that a drop is safe when analysis is incomplete.

  • Raw SQL that may reference the target is an explicit unresolved blocker.
  • Unparseable or ambiguously scoped SQL is an explicit unresolved blocker.
  • SELECT * and table.* are explicit references unless the analyzer can prove otherwise.
  • A selector pattern matching no target in any supplied model is an error, preventing typo-based false safety.
  • External application queries are outside the model and must be called out in the result/documentation.

The strongest valid conclusion is no references found in the supplied model universe, not globally safe to drop.

Suggested exit behavior:

  • 0: every selector matched and no modeled references or unresolved blockers were found
  • 1: references/blockers exist, a selector matched nothing, or model loading/analysis failed
  • 2: invalid flag combination

Acceptance criteria

  • End-to-end manifest test where a column is inherited from a base layer and referenced from different files/layers and role/environment compositions.
  • End-to-end test covering references from table expressions, projections, constraints, views, materialized views, and proxy contracts.
  • Negative test proving the same identifier on an unrelated table is not reported.
  • Wildcard, raw SQL, ambiguous query, and unparseable query tests demonstrate fail-closed output.
  • Patch/override/drop tests demonstrate that only references present in each final resolved model are attributed to that model.
  • A subprocess end-to-end test covers text and JSON output plus exit status.
  • A 30-node dump test preserves the parallel-loading guarantee and deterministic output.

Follow-up to #234 and #235.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the hclexp locate command, its existing table/column lookup from #234 and #235, and the current validator dependency graph. Use the listed acceptance tests as the implementation map, then verify deterministic manifest and dump-model results, fail-closed blockers, provenance, JSON/text output, and exit statuses.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
cli, databases, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.