[FEA] Set up infra for scheduling e2e vector database benchmarks / integration tests
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
https://github.com/rapidsai/cuvs/pull/2012 Introduces new scaffolding for running Python tests on OpenSearch backends, conditional on whether certain environment variables are defined. It's probably not likely that we'll be able to run these tests in CI, but we should establish some infrastructure so that we can still run these, maybe nightly or weekly, to make sure we're spotting and fixing issues as they arise.
It's immensely useful to be able to test these backends. We just need to make sure we're taking advantage of these tests by actually using them. Otherwise, these backends are going to get stale and start failing silently. We need to make sure our users aren't the first ones alerting us to regressions and failures.
@jrbourbeau @jameslamb @jnke2016 JFYI.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pull request #2012 and its Python test scaffolding for OpenSearch backends, focusing on the environment-variable conditions. Determine how these tests can run on a nightly or weekly schedule outside regular CI. Done means the backend integration and benchmark tests execute periodically and failures are visible to maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100