Nightly arm64 failures (2026-09-10)

Nightly arm64 run failed.

Run: https://github.com/citusdata/citus/actions/runs/34436206549

Failing jobs:
- arm64 PG18 - regress

This is the phase-1 arm64 gate (assert-off): a failure here means Citus did not
build or pass core regression/isolation tests natively on arm64.

## Investigation: reproduced on PostgreSQL 18.6 / x86_64 WSL

**Confirmed: the unchanged `pg_catalog.time_partitions` view can invoke
`time_partition_range()` on a hash partition before the range-partition filter
takes effect. A controlled execution plan reproduces the same error on x86_64,
so this failure mechanism does not require arm64.**

**Important limitation:** the ordinary local regression runs passed. The reduced
reproduction below deliberately changes session-local GEQO settings to expose a
legal, unsafe join order. This establishes the view's evaluation-order defect,
but does **not** establish which planner settings, statistics, or join order
caused the original CI failure. No execution plan was captured by that CI run.
GEQO is a diagnostic forcing mechanism here, not a claim that CI used GEQO.

### Original CI failure

- Failing job: https://github.com/citusdata/citus/actions/runs/34436206549/job/102741713636
- Citus commit: `5eaf89417fae86e60fdbfb81029292718e9bd8d5`.
- Native arm64, PostgreSQL **18.6**, assertions off.
- `check-multi` passed all 194 tests, including `multi_partitioning`.
- The subsequent `check-multi-1` failed only `multi_partitioning` among 211
tests. The later regression targets did not run because make stopped.
- Four `SELECT ... FROM time_partitions` statements unexpectedly produced:

```text
ERROR: relation "partitioning_hash_test_0" is not a range partition
DETAIL: time_partition_range can only be used for partitions of range-partitioned tables with a single partition column
```

The coordinator log locates the error in `time_partition_range`,
`src/backend/distributed/operations/partitioning.c:80`. The failing test runs on
its own line in `multi_1_schedule`; serializing that test is not an explanation
or remedy for this occurrence.

Original diff/log artifact:
https://github.com/citusdata/citus/actions/runs/34436206549/artifacts/10136453673

### Isolated local environment

- WSL2 Ubuntu 22.04, **x86_64**, GCC 11.4.0.
- PostgreSQL **18.6**, built from the upstream release archive with its SHA-256
checked. `SHOW debug_assertions` returned `off`.
- Configure options: `--enable-debug --enable-depend --disable-cassert
--enable-tap-tests --with-openssl --with-libxml --with-libxslt
--with-uuid=e2fs --with-icu --with-lz4`.
- Compiler settings: `CFLAGS=-Og -g3 -fno-omit-frame-pointer`, matching the
nightly's requested debug/optimization settings.
- Citus exported from the exact failing commit, configured with
`PG_CONFIG=/bin/pg_config --without-pg-version-check`, built
and installed with `make all` / `make install-all`.
- Installed Citus SQL version: `15.0-1`. No production C or SQL definitions were
patched during this initial reproduction phase.
- The reduced fixture uses ordinary local range/hash partitioned tables. It
needs neither distributed tables nor worker nodes.

| Experiment | Observed result |
|---|---|
| `src/test/regress/citus_tests/run_test.py multi_partitioning -b` | All 15 tests passed, including setup and `multi_partitioning`. |
| `make -C src/test/regress check-multi-1` | All 211 tests passed. |
| Reduced range/hash fixture, default planner settings | Returned the one valid range partition. |
| Same fixture, controlled GEQO settings below | Same `XX000` range-partition error in **3/3 fresh databases**. |
| Reset those settings without changing the fixture or view | Returned the valid range partition again in **3/3 databases**. |
| Original January 2021 view SELECT, evaluated as a temporary diagnostic view on the same PG18.6/current-Citus installation | Passed with defaults; reproduced the same error with the controlled settings. This was not a run of historical Citus binaries. |

### Reduced reproduction

Run with `psql -X` as a superuser in a fresh database on a Citus-preloaded
PostgreSQL 18.6 instance. These are diagnostic settings, not a proposed fix.
The expected error is allowed explicitly so psql can also run the reset control.
The three recorded runs additionally asserted that the error SQLSTATE was
`XX000`.

```sql
\set ON_ERROR_STOP on
\pset pager off
\set VERBOSITY verbose

CREATE EXTENSION citus;
CREATE SCHEMA probe8839;
CREATE TABLE probe8839.range_parent (k int) PARTITION BY RANGE (k);
CREATE TABLE probe8839.range_child
PARTITION OF probe8839.range_parent FOR VALUES FROM (0) TO (10);
CREATE TABLE probe8839.hash_parent (k int) PARTITION BY HASH (k);
CREATE TABLE probe8839.hash_child
PARTITION OF probe8839.hash_parent
FOR VALUES WITH (MODULUS 1, REMAINDER 0);

ANALYZE pg_class;
ANALYZE pg_inherits;
ANALYZE pg_partitioned_table;
ANALYZE pg_attribute;

-- Passing control: returns probe8839.range_child, with bounds 0 and 10.
EXPLAIN (VERBOSE, COSTS OFF)
SELECT parent_table, partition_column, partition, from_value, to_value
FROM time_partitions;
SELECT parent_table, partition_column, partition, from_value, to_value
FROM time_partitions;

-- Expose an alternative legal join order without changing the view.
SET geqo_threshold = 2;
SET geqo_pool_size = 2;
SET geqo_generations = 1;
SET geqo_seed = 0;

EXPLAIN (VERBOSE, COSTS OFF)
SELECT parent_table, partition_column, partition, from_value, to_value
FROM time_partitions;
\set ON_ERROR_STOP off
SELECT parent_table, partition_column, partition, from_value, to_value
FROM time_partitions;
\echo observed_sqlstate :SQLSTATE
\set ON_ERROR_STOP on

RESET geqo_threshold;
RESET geqo_pool_size;
RESET geqo_generations;
RESET geqo_seed;

-- Passing again with the exact same tables and production view.
SELECT parent_table, partition_column, partition, from_value, to_value
FROM time_partitions;
```

Controlled failure:

```text
ERROR: XX000: relation "hash_child" is not a range partition
DETAIL: time_partition_range can only be used for partitions of range-partitioned tables with a single partition column
LOCATION: time_partition_range, partitioning.c:80
```

### Captured execution-plan evidence

In the default passing plan, the `pg_partitioned_table` scan applies
`partstrat = 'r' AND partnatts = 1`, joins to the child relation, and only then
passes surviving rows into the final `Function Scan`.

The failing `EXPLAIN (VERBOSE, COSTS OFF)` has this subtree. Unrelated output
columns and intermediate joins are omitted:

```text
Nested Loop
Join Filter: (c.oid = i.inhrelid)
-> Nested Loop
-> Seq Scan on pg_catalog.pg_class c
Filter: ((c.relpartbound IS NOT NULL)
AND (relation_is_a_known_shard(c.oid) IS NOT TRUE))
-> Function Scan on pg_catalog.time_partition_range
Function Call: time_partition_range((c.oid)::regclass)
-> Nested Loop
Join Filter: (i.inhparent = p.partrelid)
...
-> Seq Scan on pg_catalog.pg_partitioned_table p
Filter: ((p.partstrat = 'r'::"char") AND (p.partnatts = 1))
...
```

The left branch calls the function directly on partition rows from `pg_class`,
including `probe8839.hash_child`, before those rows are joined to the filtered
`pg_partitioned_table` branch. Valid metadata was confirmed: the hash child has
strategy `h` and `FOR VALUES WITH (modulus 1, remainder 0)`; the range child has
strategy `r` and bounds `(0, 10)`.

The C function's direct-call contract is intentional: a direct hash-partition
call errors, while a direct range-partition call returns `0, 10`. The defect is
the view's unsafe evaluation order, not that direct invalid calls are rejected.

### Historical provenance

- #4431 introduced the view and function, merged **January 8, 2021**:
[`e7f13978b`](https://github.com/citusdata/citus/commit/e7f13978b58dede496acb6f002448c28b36893e5).
The original SELECT already calls `time_partition_range(c.oid)` in FROM and
separately filters on `p.partstrat` / `p.partnatts`.
- First shipped in [Citus 10.0.0](https://github.com/citusdata/citus/releases/tag/v10.0.0),
released February 16, 2021.
- #4725 changed an attribute-membership expression on February 18, 2021 and
shipped in 10.0.1. The original pre-change SELECT also fails in the controlled
PG18.6 experiment, so that rewrite is not required for this mechanism.
- `time_partitions/latest.sql` and `time_partition_range/latest.sql` are
unchanged between v10.0.1 and the failing HEAD. The C function's subsequent
changes only reordered includes and moved the version check.

This supports a longstanding view defect rather than a newly introduced change
to this code. It does not prove that Citus 10.0 on its contemporary PostgreSQL
versions failed in practice; no historical-binary bisect was performed.

## Fix, compatibility, and backport findings

Draft implementation: #8843, commit
[`80d88f969`](https://github.com/citusdata/citus/commit/80d88f969).

The implementation is a **production SQL-view fix**, not a test-only workaround.
In `src/backend/distributed/sql/udfs/time_partitions/latest.sql`, replace the
unguarded function argument with:

```sql
pg_catalog.time_partition_range(
CASE WHEN i.inhparent = p.partrelid
AND p.partstrat = 'r'
AND p.partnatts = 1
THEN i.inhrelid END)
```

The **parent-child equality inside CASE is necessary**. Checking only the
parent's partition strategy/count, even while referencing `i.inhrelid`, still
failed: a parameterized nested-loop plan can make a child OID available, scan
an unrelated eligible range parent, invoke the function, and only then apply
`i.inhparent = p.partrelid`. The full guard passes NULL for mismatched pairs;
the existing `STRICT` declaration prevents the C implementation from running.
The original joins and WHERE filters remain in place.

### Implementation and compatibility

- Update `time_partitions/latest.sql` and add its `15.0-1.sql` snapshot.
- Include that snapshot in the `14.0-1 -> 15.0-1` migration.
- Use `CREATE OR REPLACE VIEW pg_catalog.time_partitions` rather than dropping
the view. Its name, columns, types, owner, and object identity remain intact.
- Retain the compatible guarded definition on downgrade to `14.0-1`; its
function contract is unchanged.
- No C implementation, UDF signature, wire-format, or CI-workflow change.
Direct calls on unsupported hash/list/multi-column partitions still error;
supported range partitions retain their bounds and default-range NULLs.
- Add focused coverage to `multi_extension.sql` after the current-version
upgrade, with a matching expected-output addition. It exercises 21 GEQO
seeds and both all-column and original five-column projections. Fixtures and
planner settings are transaction-local and rolled back.
- Leave `multi_partitioning` unchanged: it also runs against N-1 SQL versions
that do not yet contain this fix. The forced-plan regression belongs in the
extension-creation/upgrade schedule, not in those older-version runs.

### Local fix evidence

| Experiment | Observed result |
|---|---|
| Full argument guard across two fixture phases, 21 seeds, and three query shapes | **126/126** exact-result comparisons passed. Covered integer/date/default range partitions, hash/list/multi-column partitions, and ordinary inheritance. Both partial guards still produced errors. |
| New `multi_extension` regression with the original regenerated, installed migration | Failed with `relation "hash_child" is not a range partition` at the forced-plan query. |
| Final patch: `run_test.py multi_partitioning -b` | **15/15** passed, including setup and the original failing test. |
| Final patch: `run_test.py multi_extension -b` | Passed, including the new 42 planner-query cases and existing extension upgrade/downgrade coverage. |
| Actual installed SQL migration: `14.0-1 -> 15.0-1 -> 14.0-1 -> 15.0-1` | Preserved the view OID, owner, ACL, extension membership, dependent user view, and expected range output. Used current binaries with version checks disabled; not a historical-binary compatibility matrix. |
| SQL snapshots | Existing snapshot consistency check passed; upgrade and downgrade files are paired. |

### Backport recommendation

Backport to the maintained **14.x** and **13.x** lines after the main fix is
accepted: `release-14.0` and `release-13.2`, respectively. The repository's
corresponding PR labels are `cherry-pick-14` and `cherry-pick-13.2`.

This is a longstanding, planner-sensitive defect rather than an ARM64-only or
PostgreSQL-18-only change. Each backport must use the target branch's actual SQL
version and appropriate migration/snapshot, preserving cross-major upgrade
paths. Do not copy the main branch's `15.0-1` wiring unchanged or rewrite released
snapshots. No backport has been implemented yet; unsupported older majors are
not routine targets.

### Remaining uncertainty and scope

The original arm64 CI trigger is still unconfirmed: its execution plan and
catalog statistics were not captured, and the exact default-settings failure
did not recur in the local targeted or full-schedule run. Native arm64 was not
rerun. The fix and migration have now been implemented and exercised locally;
native-arm64 confirmation and the broader CI matrix remain outstanding.

Full local build logs, passing/failing plans, SQL fixtures, three fresh-database
reproduction transcripts, fix comparisons, and installed-upgrade evidence are retained under
`/home/ihalatci/Development/copilot-validation/issue-8839-pg18.6/`.

Start with src/backend/distributed/sql/udfs/time_partitions/latest.sql, its 15.0-1.sql snapshot, the 14.0-1 to 15.0-1 migration, and the multi_extension.sql regression coverage. Run the focused extension upgrade test and multi_partitioning test, then verify the view preserves supported results and no longer errors under the forced planner cases.

Written by the indexing model from the issue text.