Projection pushdown ignores LIMIT BY, ARRAY JOIN and WINDOW, so columns they need get pruned
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug description
The projection pushdown pass collects which columns a query demands, then prunes the rest from
inner selects. It walks most clauses but not all three of LIMIT BY, ARRAY JOIN, and
WINDOW, so a column referenced only in one of those is invisible to the demand pass and gets
pruned out from under it.
The query then fails at ClickHouse time, with optimizeProjections on by default.
Evidence
Reference counts in posthog/hogql/transforms/projection_pushdown.py on master:
| Clause | References |
|---|---|
where |
2 |
group_by |
2 |
order_by |
2 |
having |
2 |
prewhere |
2 |
select_from |
6 |
limit_by |
0 |
array_join |
0 |
window_exprs |
0 |
Every clause that participates is referenced twice, once per phase. The three that are missing
are referenced zero times — this reads as three clauses omitted from an otherwise complete list,
not as a deliberate exclusion.
Suggested direction
Visit the three missing clauses in the demand phase, alongside the five already handled.
Notes
Static check on master today: the pass has no code path that could reach those clauses. Not
executed — a runtime repro needs ClickHouse.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in posthog/hogql/transforms/projection_pushdown.py by reading the demand phase alongside the existing where, group_by, order_by, having, and prewhere handling. Include LIMIT BY, ARRAY JOIN, and WINDOW in the same investigation, then verify that columns used only by those clauses are retained; runtime validation requires a ClickHouse repro.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100