asyncapi / asyncapi/generator

ci(workflow): Discuss selective execution of acceptance tests based on changed paths

Open
#2,134 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
1.1k
Forks
397
Avg merge
1d 7h
Merged PRs (30d)
25

Description

## Description

While reviewing the `pr-testing-with-test-project.yml` workflow, I noticed that the acceptance tests currently run for all three language templates (`js`, `py`, and `java`) whenever `should_test` is `true`, regardless of which part of the repository was modified.

After analyzing the workflow, test infrastructure, and template dependencies, I think there may be an opportunity to reduce unnecessary acceptance test executions. Before working on an implementation, I'd like to discuss whether this approach aligns with the project's expectations.

### Current behavior

The acceptance workflow uses a matrix:

```yaml
strategy:
matrix:
language: [js, py, java]
```

Once the `changes` job sets `should_test=true`, all three acceptance suites are executed.

---

### Findings from the repository analysis

#### 1. Language-specific acceptance tests are already isolated

The acceptance workflow executes one Docker Compose profile per matrix job:

* `test-js`
* `test-py`
* `test-java`

Each profile starts only its corresponding tester service.

#### 2. Shared infrastructure is independent

The workflow starts the common Microcks infrastructure using the `ci` profile before executing the language-specific profile.

The language-specific tester services do not depend on one another, only on the shared infrastructure.

#### 3. Repository has both shared and language-specific areas

From tracing the acceptance test dependency chain, there appear to be two categories of paths:

**Shared (should trigger all acceptance tests):**

* `apps/generator/**`
* `packages/helpers/**`
* `packages/components/**`
* `apps/react-sdk/**`
* Shared acceptance infrastructure (`integration-test`, `__fixtures__`, `microcks-setup`, etc.)

**Language-specific (could potentially trigger only one acceptance suite):**

* `packages/templates/clients/websocket/javascript/**`
* `packages/templates/clients/websocket/python/**`
* `packages/templates/clients/websocket/java/quarkus/**`

---

### Discussion

Would it make sense to extend the existing `paths-filter` configuration so that:

* changes to shared packages continue running all acceptance tests,
* while changes limited to a single client template execute only that client's acceptance suite?

The main goal would be to reduce unnecessary CI execution time without skipping any required validation.

---

Contributor guide

Open the contributing guide

Research direction

Start by reading pr-testing-with-test-project.yml, especially the changes job, paths-filter configuration, and language matrix. Review the ci, test-js, test-py, and test-java Docker Compose profiles and confirm the dependency paths listed in the issue. Done means an agreed design that runs only affected acceptance suites while preserving all-suite runs for shared changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, github-actions
Domain
ci-cd, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.