Unify end-to-end tests
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 34.7k
- Forks
- 2.7k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 187
Description
Is your feature request related to a problem? Please describe.
As a new contributor, I find the process of running all of the end-to-end tests to be complicated. There are a bunch of script files, and the way they are run is inconsistent:
tests/snapshots/snapshots-recovery.sh: buildsqdrantdocker image and tests it.tests/storage-compat/storage-compatibility.sh: builds./target/debug/qdrantusingcargoand runs it.tests/basic_*_test.sh: callscurl | jq/gpcurlto already running qdrant instance. They don't check the result, just print responses to stdout.#3606tests/openapi_integration_test.sh: runs all pytest files fromopenapi/tests/openapi_integrationusing dockerized pytest. Hasrequirements.txtandDockerfileto build pytest docker image.tests/consensus_tests: pytest tests. Buildsqrantdocker image and runs it via docker-compose.tests/shard-snapshot-api.sh: is not a test but rather a helper tool to call an API.tests/grpc_consistency_check.sh: check that autogenerated files are up to date.
Also, it's not clear where I am supposed to add new tests and which language to use (bash or python).
Describe the solution you'd like
It would be helpful if there were a single command to run all of the end-to-end tests or to run specific tests (akin to cargo test -- --test TEST_NAME or pytest TEST_FILE.py::TEST_NAME).
It should be possible to run tests without building a qdrant docker image since the building process is quite slow.
As for grpc_consistency_check.sh, I feel that its invocation should be included in a generic task runner script (e.g. make gen, alongside make e2e-tests, make fmt, and make lint). This would make pre-push checks easier.
Describe alternatives you've considered
Add a list of test files with an instruction on how to run them to the documentation. However, the documentation tends to become outdated over time as it's easy to forget to update it when changing the test or adding a new one.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inventorying the existing entry points under tests/, including snapshots-recovery.sh, storage-compatibility.sh, basic_*_test.sh, consensus_tests, and grpc_consistency_check.sh. Compare how each test is built and run, then define a consistent command structure and document how specific tests run. Done means the end-to-end tests have a unified, maintainable entry point and the generated-file check is included in the appropriate task workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, python, rust, shell
- Domain
- build-system, developer-experience, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100