KhronosGroup / KhronosGroup/OpenCL-CTS

Reducing CTS runtime

Open
#2,723 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
232
Forks
235
Avg merge
8d 7h
Merged PRs (30d)
18

Description

This issue explains all the changes I've made to optimize the OpenCL-CTS to reduce the time it takes to run by more than 10x (up to 20x depending on the actual driver/device).

The overall changes can be found in [my branch](https://github.com/rjodinchr/OpenCL-CTS/tree/main)

----------------------------------------------------------
## `test_common/harness` & CI
### Special Values Management
* Group float and double special values into a single file (`conversions.cpp`).
* Expose `GetFpSpecialValues` (for floats) and `GetIntSpecialValues` (for integers) in `conversions.h`.
* Implement a static buffer with a mutex to cache computations, ensuring thread-safe concurrent access.

### Test Execution & Parsing
* Rename `parseCustomParam` to `parseCommonParamAndGetRemovedArgs` to better reflect its actual behavior.
* Add `removed_args` and `help` as output parameters to improve usage clarity and track function actions later in the flow.
* Introduce a `ParseArgsFn` parameter to `runTestHarnessWithCheckAndParse` to reduce boilerplate code while retaining necessary data for JSON generation.
* Clean up memory management by removing excessive C-style `malloc`, `calloc`, and `free` operations.
* Fix parsing issues for previously unreachable arguments.

### Registry & Output
* Remove `testNum` and `testList` from `runTestHarness*` functions to retrieve them directly from the test registry instead.
* Update `test_registry` templates to remove unnecessary private fields and classes, streamlining dynamic registration.
* Refine the test listing behavior (`--list`) to only print subtests when explicitly requested, reducing visual clutter.
* Implement `cl_channel_type_to_string` for finer-grained image sub-tests to improve parallelization performance.

### CI
* Update `saveResultsToJson` to include meaningful arguments, making it easier for dynamic CI testing to track expected outcomes.
* Update CI comparison scripts (`compare_results.py`) to validate against specific execution arguments recorded in `golden.json`.

----------------------------------------------------------
## Tests Only Modified to Match New Harness

Updated the following tests solely to match the new harness:
`SVM`, `allocations`, `api`, `atomics`, `basic`, `buffers`, `c11_atomics`, `commonfns`, `compiler`, `computeinfo`, `contractions`, `d3d10`, `d3d11`, `device_execution`, `device_partition`, `device_timer`, `events`, `extensions/cl_ext_buffer_device_address`, `extensions/cl_ext_cxx_for_opencl`, `extensions/cl_khr_command_buffer/cl_khr_command_buffer_mutable_dispatch`, `extensions/cl_khr_command_buffer`, `extensions/cl_khr_dx9_media_sharing`, `extensions/cl_khr_external_memory_ahb`, `extensions/cl_khr_external_memory_dma_buf`, `extensions/cl_khr_external_semaphore`, `extensions/cl_khr_external_semaphore_dx_fence`, `extensions/cl_khr_kernel_clock`, `extensions/cl_khr_semaphore`, `generic_address_space`, `geometrics`, `gl`, `gles`, `mem_host_flags`, `multiple_device_context`, `non_uniform_work_group`, `pipes`, `printf`, `profiling`, `relationals`, `spirv_new`, `subgroups`, `thread_dimensions`, `vulkan`, `workgroups`.

----------------------------------------------------------
## `test_conformance/half`
### Data Inputs & Special Values
* Reduce the number of inputs to 2^26.
* Utilize `GetFpSpecialValues` supplemented with randomly generated numbers.

----------------------------------------------------------
## `test_conformance/select`
### Execution Flow & Input Generation
* Enqueue static inputs (`src1` & `src2`) only once at the beginning.
* Test all value combinations per vector for the `cmpbuffer` instead of evaluating all 2^32 combinations (e.g., resulting in 2^16 inputs for 16-element vectors).
* Force the MSB on random numbers for vectors to evaluate them as strictly true or false.
* Alternate between zero and non-zero values for scalar inputs.

----------------------------------------------------------
## `test_conformance/vectors`
### Refactoring & Performance
* Refactor the code to parallelize the execution of alignment tests.
* Increase the buffer size to 8MB to extend compute time and improve overall test efficiency.

----------------------------------------------------------
## `test_conformance/conversions`
### Data Inputs & Special Values
* Limit standard datasets to 2^25 elements while adding the `-a` option for exhaustive 2^32 element testing.
* Retain wimpy/embedded modes for reduced workloads.
* Replace hardcoded special values with `GetFpSpecialValues` supplemented by random numbers.

### Parallelization & Execution Flow
* Utilize dynamic test registration to be able to execute them in parallel. It also allows one to use wildcards to run a subset.
* Implement a buffer pool to eliminate global variables, enabling safe parallel runs.
* Simplify execution by replacing complex `cl_event` chains with a straightforward write/compute/read pattern.
* Copy inputs once for all vector sizes.
* Trigger verification immediately using `cl_event` on read operations for a given vector size.
* Increase buffer size to 8MB for better computational efficiency.
* Refactor the types iterator for a cleaner, more concise pattern.

----------------------------------------------------------
## `test_conformance/integer_ops`
### Data Inputs & Verification
* Reduce dataset sizes based on bit-width: 2^25 (64-bit), 2^24 (32-bit), 2^23 (16-bit), and 2^16 (8-bit).
* Replace `quick*` tests with wimpy mode (reducing dataset sizes while still testing all special value combinations).
* Use `GetIntSpecialValues` combined with randomly generated numbers.
* Optimize the verification process by using `memcmp` before performing element-wise comparisons.

### Execution Flow & Performance
* Dynamically adjust buffer size during multi-threaded execution to prevent exceeding global device memory limits.
* Increase buffer size to 8MB to ensure sufficient computation for good efficiency.
* Copy input values once for all vector sizes.
* Enqueue the NDRange and `readbuffer` for all vector sizes, utilizing `cl_event` on the final read of each vector size.
* Iterate over vector sizes prior to iterating over buffer elements.
* Wait on the `readbuffer` `cl_event` for each vector size to begin its corresponding verification.

----------------------------------------------------------
## `test_conformance/math_brute_force`
### Data Inputs
* Reduce standard dataset size to 2^27.
* Add an `-a` option for exhaustive 2^32 element testing.
* Retain wimpy/embedded mode and use special values supplemented by randomly generated numbers.

### Execution Flow & Parallelization
* Refactor code to trigger verification using `cl_event` on read/map buffers.
* Invert the element/vectorSize loop to enhance performance, particularly concerning `clwaitevent`.
* Implement a resource pool to prevent repeated allocation/deallocation and eliminate global variables.
* Protect remaining global variables and `ThreadPool` with mutexes for safe parallel execution.
* Increase buffer size to 8MB to optimize test efficiency.

----------------------------------------------------------
## `test_conformance/images`
### Execution Flow & Test Granularity
* Consolidate image initialization and copying logic into reusable helper functions (`test_copy_init_images`, `test_copy_image_generic`).
* Replace global variables with structures passed between functions to enable parallel execution.
* Write the input buffer only once in `cl_copy_images` to remove redundant copies.
* Utilize dynamic test registration to create finer-grained tests, improving parallelization performance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked branch and the changes under test_common/harness, then inspect the affected test_conformance areas and compare_results.py. Done means the harness and listed tests support the described parallel execution and workload changes, CI result comparisons remain valid, and runtime is reduced as reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ci-cd, performance, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.