asyncapi / asyncapi/cli

[BUG] `npm test` is non-hermetic: integration tests trigger real npm installs and fail due to stdout/stderr pollution

Open
#2,004 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
272
Forks
375
Avg merge
3h 22m
Merged PRs (30d)
8

Description

## Describe the bug

Running `npm test` in the AsyncAPI CLI repository fails on a clean or typical local setup, even when the CLI code itself has not been changed.

A large number of integration tests fail because some CLI commands trigger **real npm dependency resolution / installation steps** during test execution. This causes npm to print progress output (for example `idealTree` spinners and dependency logs) to `stdout`.

Many tests expect:
- empty output (`''`), or
- an exact, fixed string

Because of this, the additional npm output breaks assertions and causes widespread test failures that are **not related to actual CLI behavior or regressions**.

---

## Current behavior

- `npm test` produces many failing integration tests.
- Failures contain unexpected output such as:
- `idealTree:*`
- npm progress spinners
- dependency resolution logs
- Test results vary depending on:
- npm version
- local environment
- network conditions
- OS setup

This makes the test suite **non-deterministic** and hard to trust.

Image

---

## Expected behavior

- `npm test` should pass on a clean local setup without requiring special environment tweaks.
- CLI integration tests should:
- not rely on real npm installs, or
- mock / silence npm output during tests, or
- assert exit codes and behavior instead of exact stdout content.
- Test results should be deterministic and environment-independent.

---

## Steps to reproduce

1. Clone the repository:
```bash
git clone https://github.com/asyncapi/cli.git
cd cli
```
2. Install dependencies:
```
npm install
```
3. Run the test suite:
```
npm test
```
4. Observe:

- Multiple integration test failures
- Unexpected npm output (idealTree, spinners, logs) appearing in assertions
- Tests failing even though CLI logic is unchanged

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.