Cloud-CV / Cloud-CV/EvalAI

Integrate Postman API Tests into GitHub Actions CI Pipeline

Open
#4,677 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
984
Avg merge
2h 54m
Merged PRs (30d)
14

Description

**Description:**

To improve backend API reliability, we should integrate **Postman API tests** into EvalAI’s **GitHub Actions CI pipeline**. This will ensure that core API endpoints remain functional and are not unintentionally broken by future pull requests — especially since current CI primarily validates backend unit tests and frontend builds.

This was originally proposed for Travis CI, but since we are migrating to **GitHub Actions**, the integration should be done there.

**Plan:**

### ✅ Step 1: Create Postman Test Assets

* [ ] Create a folder at the root of the repo (e.g., `postman-tests/`)
* [ ] Add the following files:

* `evalai-api-tests.postman_collection.json`: Contains the API test cases.
* `evalai-env.postman_environment.json`: Holds base URL, auth tokens, etc.

### ✅ Step 2: Add GitHub Actions Workflow

* [ ] Use a GitHub Actions workflow to:

* Set up Node.js
* Install [[Newman](https://github.com/postmanlabs/newman)](https://github.com/postmanlabs/newman), the CLI tool for Postman
* Run the collection:

```bash
newman run postman-tests/evalai-api-tests.postman_collection.json \
-e postman-tests/evalai-env.postman_environment.json
```
* Fail the build if any API test fails

**Additional Notes:**

* This should be coordinated with or follow the [[Travis → GitHub Actions migration](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576).

Contributor guide

Open the contributing guide

Research direction

Review the Travis-to-GitHub Actions migration in pull request #4576, then define the API cases and environment values for postman-tests/evalai-api-tests.postman_collection.json and postman-tests/evalai-env.postman_environment.json. Add a GitHub Actions workflow that sets up Node.js, installs Newman, and runs the collection; done means the workflow fails when an API test fails and passes for the supported endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, node.js, postman
Domain
api, ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.