KhronosGroup / KhronosGroup/OpenCL-CTS

Extensions Testing and Conformance Verification

Open
#2,750 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
232
Forks
235
Avg merge
8d 7h
Merged PRs (30d)
18

Description

## The Problem

The current OpenCL conformance process relies on running tests listed in CSV files (such as `opencl_conformance_tests_full.csv`), as described in the [generating a conformance report](https://github.com/KhronosGroup/OpenCL-CTS#generating-a-conformance-report) section of the README.

This approach has several limitations:
1. **Outdated Conformance Lists:** The CSV files often lag behind, omitting tests for newly added extensions.
2. **Maintenance Overhead:** Updating and maintaining these CSV files manually is awkward and error-prone (refer to discussions in PR #2096).
3. **Build-Time vs. Runtime Gating:** Many extension tests are gated behind CMake flags and are not built by default. This makes it difficult for implementers to know what needs to be run to claim full conformance, and complicates the build process.
4. **Poor Discoverability:** It is not obvious to external implementers which extension tests exist and should be run.

## Proposed Solutions / Discussion Points

We propose addressing these issues by moving away from build-time gating and CSV-based test selection, towards consolidated binaries and runtime feature detection.

### 1. Consolidate Test Binaries
* **One CTS Binary:** Ideally, collapse the entire Conformance Test Suite into a single executable.
* **One Extensions Binary:** As an intermediate step, consolidate all extension tests into a single executable, separate from the core CTS binary.

### 2. Always Build, Runtime Skip (with Stub Implementations)
Instead of hiding extension tests behind CMake flags:
* **Always build all extension tests** by default.
* **Runtime Skipping:** Tests should dynamically query the device (e.g., via `CL_DEVICE_EXTENSIONS` or `CL_DEVICE_EXTENSIONS_WITH_VERSION`) to check if the extension is supported. If not supported, the test should be gracefully skipped.
* **Stub Implementations:** To facilitate building tests for extensions that might not be supported by the host OpenCL loader/runtime at link time, we should employ stub implementations.
* If a test is run for an unsupported extension, it must skip.
* If the test fails to skip and attempts to call the extension API through the stub, the stub implementation should ensure a clear error is returned (and the test fails), rather than crashing or silently passing.

### 3. Long-Term Direction: dEQP-like Architecture
* Consider a long-term migration towards a dEQP-like framework. This would involve a single runner with dynamic test discovery and filtering capabilities, eliminating the need for maintaining static CSV files for conformance runs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README's generating-a-conformance-report section, the opencl_conformance_tests_full.csv list, and the CMake options that gate extension tests. Map how extension tests are built and selected, then compare that flow with the proposed consolidated binary and runtime feature checks. Done means the project has an agreed implementation plan for always-built extension tests, safe unsupported-extension handling, and reduced reliance on CSV selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.