Integrate Postman API Tests into GitHub Actions CI Pipeline
- Dominant language
- Python
- Stars
- 2k
- Forks
- 984
- Avg merge
- 2h 54m
- Merged PRs (30d)
- 14
Description
**Description:**
To improve backend API reliability, we should integrate **Postman API tests** into EvalAI’s **GitHub Actions CI pipeline**. This will ensure that core API endpoints remain functional and are not unintentionally broken by future pull requests — especially since current CI primarily validates backend unit tests and frontend builds.
This was originally proposed for Travis CI, but since we are migrating to **GitHub Actions**, the integration should be done there.
**Plan:**
### ✅ Step 1: Create Postman Test Assets
* [ ] Create a folder at the root of the repo (e.g., `postman-tests/`)
* [ ] Add the following files:
* `evalai-api-tests.postman_collection.json`: Contains the API test cases.
* `evalai-env.postman_environment.json`: Holds base URL, auth tokens, etc.
### ✅ Step 2: Add GitHub Actions Workflow
* [ ] Use a GitHub Actions workflow to:
* Set up Node.js
* Install [[Newman](https://github.com/postmanlabs/newman)](https://github.com/postmanlabs/newman), the CLI tool for Postman
* Run the collection:
```bash
newman run postman-tests/evalai-api-tests.postman_collection.json \
-e postman-tests/evalai-env.postman_environment.json
```
* Fail the build if any API test fails
**Additional Notes:**
* This should be coordinated with or follow the [[Travis → GitHub Actions migration](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576).
Contributor guide
Research direction
Review the Travis-to-GitHub Actions migration in pull request #4576, then define the API cases and environment values for postman-tests/evalai-api-tests.postman_collection.json and postman-tests/evalai-env.postman_environment.json. Add a GitHub Actions workflow that sets up Node.js, installs Newman, and runs the collection; done means the workflow fails when an API test fails and passes for the supported endpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, node.js, postman
- Domain
- api, ci-cd, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100