ga4gh / ga4gh/openapi-test-runner

Add TES conformance tests for CANCELING lifecycle and cancel idempotency

Open
#80 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
HTML
Stars
5
Forks
6
PR merge metrics
No merged PRs in 30d

Description

### Summary
The TES conformance suite currently does not enforce behavior around the CANCELING state during task cancellation. CANCELING is part of TES state modeling, but there is no dedicated conformance coverage to verify that cancellation is represented as an in-progress lifecycle before terminal CANCELED.

### Problem
Without explicit tests, implementations may:
- jump directly to CANCELED before cleanup actually completes
- report terminal cancellation while compute resources are still being cleaned up
- handle repeated cancel requests inconsistently

This creates gaps in conformance and can hide lifecycle/state transition bugs.

### Proposed test coverage
Add TES conformance tests (for both 1.0.0 and 1.1.0 profiles where applicable) that validate:

1. Cancel transitions through CANCELING
- Create a task expected to remain active long enough to cancel
- Call cancel endpoint
- Verify subsequent task reads can report CANCELING while cancellation is in progress

2. Eventual terminal state
- Poll task status after cancel request
- Verify task eventually reaches CANCELED within timeout

3. Idempotent re-cancel behavior
- Call cancel again when task is already CANCELING or CANCELED
- Verify behavior is non-error/idempotent and does not violate response schema

4. Schema conformance
- Ensure task state values observed during this flow validate against TES schema models, including CANCELING and CANCELED

5. Optional list-task visibility check
- Validate that list/read endpoints remain schema-valid and coherent while task is in CANCELING

### Why this belongs in openapi-test-runner
This is conformance behavior that should be validated across implementations, not only in individual server repos. Adding this test ensures lifecycle correctness is verified consistently in compliance runs.

#### Acceptance criteria
- New conformance test scenario(s) added for TES cancel lifecycle
- Tests verify CANCELING to CANCELED transition semantics with polling
- Tests verify repeated cancel calls do not break conformance
- Tests run under existing report flow and produce pass/fail results like other TES tests
- Documentation updated to describe expected cancel lifecycle behavior being tested

Potential implementation note
Because cancellation cleanup timing differs by backend, tests should allow bounded eventual consistency (poll with timeout) instead of requiring immediate CANCELED.

Proposed new file: tests/cancel_task_lifecycle.yml

```
description: Job to validate CANCELING -> CANCELED lifecycle
service: TES
versions:
- 1.0.0
- 1.1.0
tags: []
jobs:
- name: create_task
description: Create a long-running TES task
endpoint: /tasks
operation: POST
request_body: |
{
"name": "CancelLifecycleTest",
"description": "Task intentionally sleeps so cancel can be observed",
"executors": [
{
"image": "alpine",
"command": ["sh", "-c", "sleep 180"]
}
]
}
storage_vars:
id: $response.id
response:
200: {}

- $ref: "./templates/cancel_task_template.yml"

- name: get_task
description: Read task immediately after cancel request
endpoint: /tasks/{id}
operation: GET
path_parameters:
id: "{id}"
query_parameters:
- view: MINIMAL
filter:
- path: $response.state
type: string
value: CANCELING
response:
200: {}

- name: get_task
description: Poll until task reaches terminal canceled state
endpoint: /tasks/{id}
operation: GET
path_parameters:
id: "{id}"
query_parameters:
- view: MINIMAL
polling:
interval: 10
timeout: 3600
env_vars:
check_cancel: True
filter:
- path: $response.state
type: string
value: CANCELED
response:
200: {}
```

Proposed new file: tests/cancel_task_idempotent.yml

```
description: Job to validate cancel idempotency
service: TES
versions:
- 1.0.0
- 1.1.0
tags: []
jobs:
- name: create_task
description: Create a long-running TES task
endpoint: /tasks
operation: POST
request_body: |
{
"name": "CancelIdempotencyTest",
"description": "Task intentionally sleeps so repeated cancel can be tested",
"executors": [
{
"image": "alpine",
"command": ["sh", "-c", "sleep 180"]
}
]
}
storage_vars:
id: $response.id
response:
200: {}

- $ref: "./templates/cancel_task_template.yml"

- name: cancel_task
description: Re-cancel the same TES task (should be no-op/idempotent)
endpoint: /tasks/{id}:cancel
operation: POST
path_parameters:
id: "{id}"
response:
200:

- name: get_task
description: Poll until task reaches terminal canceled state
endpoint: /tasks/{id}
operation: GET
path_parameters:
id: "{id}"
query_parameters:
- view: MINIMAL
polling:
interval: 10
timeout: 3600
env_vars:
check_cancel: True
filter:
- path: $response.state
type: string
value: CANCELED
response:
200: {}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed tests/cancel_task_lifecycle.yml and tests/cancel_task_idempotent.yml, then read templates/cancel_task_template.yml and the existing report flow they use. Run comparable TES conformance jobs first. Done means both profiles cover CANCELING to CANCELED polling, repeated cancel requests, and schema-valid responses, with the expected lifecycle behavior documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi, yaml
Domain
api, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.