eclipse-tractusx / eclipse-tractusx/bpdm

CICD: System Test Execution Reporting On Jira

Open
#1,683 1 comment 0 reactions 0 assignees View on GitHub
github_actions
Dominant language
Kotlin
Stars
12
Forks
28
Avg merge
3d 14h
Merged PRs (30d)
41

Description

# Integrate Chart Test Results with Jira Xray for Documentation

## Background

The `bpdm-system-tester` module runs end-to-end chart tests using Cucumber as its execution engine, reading Gherkin `.feature` files to exercise the full BPDM service stack (Gate → Orchestrator → Pool). Currently, test results exist only as console output and are not persisted or visible in Jira for traceability, documentation, or stakeholder review.

## Goal

Wire the chart test execution results into Jira Xray so that every scenario run in CI/CD is automatically reflected in a corresponding Jira test issue, producing an auditable test execution record.

---

## Xray Data Model

Xray introduces three issue types relevant to this integration:

- **Test** — one issue per Gherkin scenario; holds the scenario definition and its execution history
- **Test Execution** — created per CI run; links to all Test issues executed in that run with their PASS/FAIL status
- **Test Plan** — optional grouping of Test Executions by release or environment

Each CI run will auto-create a new Test Execution issue and link it to the corresponding Test issues. Whether Test Executions should additionally be grouped under a Test Plan is an open point (see below).

---

## Tasks

### 1. Create Jira Test Issues

For every Gherkin scenario in the `.feature` files, create a corresponding **Test** issue in Jira Xray. Each issue represents one scenario and will hold its execution history.

Affected feature files:
- `src/main/resources/cucumber/share_generic_business_partner.feature`
- `src/main/resources/cucumber/share_business_partner_relation.feature`

The number of Test issues to create for Scenario Outlines depends on the tagging strategy chosen (see Open Points).

### 2. Tag Feature File Scenarios with Jira Issue Keys

Add Jira Xray tags to each scenario (and scenario outline) in the `.feature` files so the Cucumber–Xray integration can link execution results to the correct Jira Test issues.

Example:
```gherkin
@BPDM-1234
Scenario: Share Without Address Type
Given ...
```

All scenarios in both feature files need a unique `@` tag. The tagging strategy for Scenario Outlines must be decided before creating the Jira issues (see Open Points).

### 3. Add HTTP Request/Response Logging

The system tester's API clients (`GateClientConfig`, `PoolClientConfig`, `OrchestratorClientConfig`) currently have no structured logging of outgoing requests or incoming responses. Add logging so that a human reading a CI/CD log can follow exactly what was sent and received during each scenario execution.

Implementation guidance:
- Add a logging interceptor (e.g. a Spring `ClientHttpRequestInterceptor`) to each RestClient/Feign client configured in the `config/` classes
- Log at minimum: HTTP method, URL, request body, response status, and response body
- Log at the `DEBUG` level; enable it via a configuration property so it can be turned on in CI without changing code

### 4. Enable Cucumber JSON Report Output

Jira Xray import requires a structured test report in **Cucumber JSON format** — this is the native format produced by Cucumber's `--plugin json:` flag. Do not use JUnit XML: it strips all Cucumber metadata (tags, feature file structure, scenario names) that Xray needs to map results back to Test issues. Do not confuse Cucumber JSON with Xray's own proprietary "Xray JSON" format — these are different schemas uploaded to different API endpoints.

In `Application.kt`, add the reporting plugin to the default Cucumber arguments passed to `io.cucumber.core.cli.Main`:

```kotlin
fun main(args: Array) {
var cucumberArgs = if (!args.contains("--threads")) args.plus(listOf("--threads", "16")) else args
cucumberArgs = if (!args.contains("--plugin")) cucumberArgs.plus(listOf("--plugin", "json:target/cucumber-reports/cucumber.json")) else cucumberArgs
io.cucumber.core.cli.Main.main(*cucumberArgs)
}
```

The guard on `--plugin` ensures callers can override the report path without producing duplicate plugins.

### 5. Upload Test Results to Jira Xray in the CI/CD Pipeline

After the chart tests complete, add a pipeline step that uploads `target/cucumber-reports/cucumber.json` to Xray using the **Cucumber-specific import endpoint** (not the generic Xray JSON endpoint).

**Critical — handling test failures in CI:**
`io.cucumber.core.cli.Main.main()` calls `System.exit(1)` when any scenario fails, terminating the JVM before the pipeline reaches the upload step. To guarantee the upload always runs:

- Replace `Main.main()` with `Main.run()` in `Application.kt`, which returns the exit code as a `byte` without calling `System.exit`. Propagate the exit code explicitly after the upload completes:

```kotlin
fun main(args: Array) {
var cucumberArgs = if (!args.contains("--threads")) args.plus(listOf("--threads", "16")) else args
cucumberArgs = if (!args.contains("--plugin")) cucumberArgs.plus(listOf("--plugin", "json:target/cucumber-reports/cucumber.json")) else cucumberArgs
val exitCode = io.cucumber.core.cli.Main.run(*cucumberArgs)
// upload report here (or delegate to CI pipeline step after this process exits)
System.exit(exitCode.toInt())
}
```

- Alternatively, keep `Main.main()` and use `continue-on-error: true` on the test step combined with `if: always()` on the upload step in the CI pipeline YAML. This is simpler but relies on correct pipeline configuration.

**Upload details:**
- The API endpoint, authentication mechanism, and available GitHub Actions differ between Xray Cloud and Xray Server/DC (see Open Points) — these must be confirmed before implementing this step
- Use a scoped service-account token stored as a CI/CD secret
- Each upload automatically creates a new **Test Execution** issue in Jira and links results to the corresponding Test issues

---

## Acceptance Criteria

- [x] Every Gherkin scenario in both `.feature` files has a corresponding Jira Xray Test issue
- [x] Each scenario is tagged with its Jira issue key (`@`)
- [ ] CI/CD pipeline produces a Cucumber JSON report (`target/cucumber-reports/cucumber.json`) after each run
- [x] HTTP requests and responses made during test execution are logged at `DEBUG` level and visible in CI/CD logs when enabled
- [ ] The pipeline uploads the report to Jira Xray after each run and the linked Jira Test issues show updated execution status (PASS/FAIL)
- [ ] Test failures do not prevent the upload step from running

---

## Open Points

These questions must be answered before or during implementation, as the answers affect scope and design.

**OP-1 — Xray Cloud vs. Xray Server/Data Center**
Which Xray variant is in use? The two products have different APIs, different authentication mechanisms, and different GitHub Actions support:

| | Xray Cloud | Xray Server / Data Center |
|---|---|---|
| Auth | Client ID + Secret → JWT token | Basic auth or PAT |
| Cucumber import endpoint | `POST /api/v2/import/execution/cucumber` | `POST /rest/raven/1.0/import/execution/cucumber` |
| GitHub Action | `mikepenz/xray-action` | Manual `curl` or third-party |

This decision drives almost every detail of task 5.

**OP-2 — Scenario Outline tagging strategy**
The two feature files contain Scenario Outlines with multiple example rows. Xray supports two approaches and they produce different numbers of Jira issues:

- **One tag per Outline** — Xray treats each example row as an iteration of a single Test issue. Fewer issues, simpler setup, but less granular: a failure in one row fails the whole Test issue.
- **One tag per example row** — each row maps to its own Test issue. More issues to create, but each row is independently trackable.

Xray's behavior for this also differs between Cloud and Server. This must be decided before creating the Test issues in task 1.

**OP-3 — Test Plan grouping**
Should Test Executions (auto-created per CI run) be grouped under a standing **Test Plan** issue in Jira? A Test Plan allows filtering execution history by release or environment. If yes, a Test Plan issue needs to be created and its key needs to be passed in the upload request.

---

## Notes

- The system tester uses Cucumber 7.34.3; Xray's Cucumber import is compatible with the standard Cucumber JSON format produced by this version
- Cucumber 7's JSON formatter is synchronized and thread-safe, so parallel execution with 16 threads will produce a single correct `cucumber.json` file; scenario entries will appear in completion order rather than feature-file order, which Xray handles correctly
- The `no-auth` Spring profile is available for local validation without Keycloak

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.