Altinity / Altinity/clickhouse-regression

Feature-support validation: fail fast on missing Antalya features on antalya-26.6 (swarms, oauth, s3 export, iceberg)

Open
#155 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

26.6-antalya cicd-failure
Dominant language
Python
Stars
27
Forks
10
Avg merge
1m
Merged PRs (30d)
2

Description

This issue records work done while running regression suites against the new antalya-26.6 base. Antalya features are forward-ported to a new base at different times, so several suites failed en masse because the features they test are not present yet. The work was consolidated into a single commit, plus two follow-up commits (see below). Assisted by AI (Cursor / Claude Opus 4.8).

Summary

On antalya-26.6, suites that exercise not-yet-forward-ported Antalya features produced large numbers of unrelated failures (e.g. swarms had ~3000+ failures, all rooted in one missing setting), which makes triage hard and hides real regressions.

Added a reusable feature-support validation so each affected suite/sub-feature emits one clear failure (a feature support validation subtest) instead of many, and does not run the rest once the feature is known to be missing. Gates are antalya-only where applicable, so non-antalya behavior is unchanged, and they auto-recover once the feature is forward-ported (the check passes and the tests run normally).

Fix commit: fb33c1f4c

Generic component

helpers/feature_support.py:

  • feature_support_validation — a @TestScenario named feature support validation that fails with a clear message when the feature is absent.
  • validate_feature_support(self, feature, check) — runs the detection on the module context and the named subtest; returns the support bool so a suite can return/skip the rest.
  • setting_supported(name) — check that a setting exists (queries the name column of system.settings, not its value — several settings default to an empty string, so a value-based check gives false negatives).
  • get_clickhouse_binary / run_clickhouse_local — pre-cluster host probe via clickhouse local, for features that stop the server from starting.

Per-suite gates

Suite / sub-feature Detection Notes
swarms object_storage_cluster setting collapsed ~3000+ failures into one
oauth CREATE USER ... IDENTIFIED WITH jwt via clickhouse local (pre-cluster) server can't even start without JWT (static jwt_user in users.xml is rejected at boot)
s3 export part allow_experimental_export_merge_tree_part
s3 export partition export_merge_tree_partition_force_export partition-specific, not the general export-part setting
iceberg swarm object_storage_cluster
iceberg iterator race condition object_storage_cluster_join_mode
iceberg sort key timezone iceberg_partition_timezone
iceberg export partition export_merge_tree_partition_force_export server crashes on config restart without it; gate runs before applying the config
iceberg system.tables partition/sorting keys capability probe (partition_key populated in system.tables) PR Altinity/ClickHouse#1432 — no setting exists, so we probe the actual capability

Detection notes

  • Most features are gated by a setting existence check (post-cluster query, or pre-cluster clickhouse local for oauth).
  • system.tables partition/sorting keys has no setting (compiled-in behavior from PR #1432, "Antalya 26.1"), so it uses a capability probe; the test's own version gate (>26.3.10.20001) wrongly assumed presence on 26.6.
  • Confirmed detection empirically against a build that has the features (26.3.10.20001.altinityantalya, passes) vs antalya-26.6 (fails fast).

Follow-up commits

  • 6933926130 — all setting-based gates now call setting_supported() directly. Removes three duplicate wrappers (export_merge_tree_part_supported, export_merge_tree_partition_supported, object_storage_cluster_supported) and swarms/tests/feature_support.py. From Julian's review: the same check for export_merge_tree_partition_force_export had been written two different ways across the s3 and iceberg suites.
  • 867b4caf2f — fixes a TypeError this commit introduced in swarms/regression.py. validate_feature_support() ends with a Scenario, so the with And(...) below it had no Step sibling to inherit its subtype from. It only triggers when the gate passes, so the swarms suite did not actually run on a swarm-capable build until this fix — the "tests run normally" part of the summary above only holds from 867b4caf2f on.

Follow-ups (not addressed here)

  • None

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The described work is already implemented across helpers/feature_support.py and the swarms, oauth, s3, and iceberg suites; start by reviewing commits fb33c1f4c, 6933926130, and 867b4caf2f. Run the affected regression suites against a feature-capable build and antalya-26.6 to verify that missing features produce one validation failure while supported features run normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.