qdrant / qdrant/qdrant

Unify end-to-end tests

Open
#3,382 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted core
Dominant language
Rust
Stars
34.7k
Forks
2.7k
Avg merge
1d 18h
Merged PRs (30d)
187

Description

Is your feature request related to a problem? Please describe.
As a new contributor, I find the process of running all of the end-to-end tests to be complicated. There are a bunch of script files, and the way they are run is inconsistent:

  • tests/snapshots/snapshots-recovery.sh: builds qdrant docker image and tests it.
  • tests/storage-compat/storage-compatibility.sh: builds ./target/debug/qdrant using cargo and runs it.
  • tests/basic_*_test.sh: calls curl | jq / gpcurl to already running qdrant instance. They don't check the result, just print responses to stdout.
  • tests/openapi_integration_test.sh: runs all pytest files from openapi/tests/openapi_integration using dockerized pytest. Has requirements.txt and Dockerfile to build pytest docker image. #3606
  • tests/consensus_tests: pytest tests. Builds qrant docker image and runs it via docker-compose.
  • tests/shard-snapshot-api.sh: is not a test but rather a helper tool to call an API.
  • tests/grpc_consistency_check.sh: check that autogenerated files are up to date.

Also, it's not clear where I am supposed to add new tests and which language to use (bash or python).

Describe the solution you'd like
It would be helpful if there were a single command to run all of the end-to-end tests or to run specific tests (akin to cargo test -- --test TEST_NAME or pytest TEST_FILE.py::TEST_NAME).
It should be possible to run tests without building a qdrant docker image since the building process is quite slow.

As for grpc_consistency_check.sh, I feel that its invocation should be included in a generic task runner script (e.g. make gen, alongside make e2e-tests, make fmt, and make lint). This would make pre-push checks easier.

Describe alternatives you've considered
Add a list of test files with an instruction on how to run them to the documentation. However, the documentation tends to become outdated over time as it's easy to forget to update it when changing the test or adding a new one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying the existing entry points under tests/, including snapshots-recovery.sh, storage-compatibility.sh, basic_*_test.sh, consensus_tests, and grpc_consistency_check.sh. Compare how each test is built and run, then define a consistent command structure and document how specific tests run. Done means the end-to-end tests have a unified, maintainable entry point and the generated-file check is included in the appropriate task workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, python, rust, shell
Domain
build-system, developer-experience, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.