brain-score / brain-score/vision

Parallel test execution causes race conditions in the shared test database

Open
#422 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

Current endpoint unit tests and end-to-end integration tests all run on a single database. Before and after each test, the database is emptied to create isolated conditions for testing. An example of this is [test_endpoints.py](https://github.com/brain-score/brain-score/blob/integrate_core/tests/test_submission/test_endpoints.py).

However, when multiple tests that access the database run in parallel (e.g. tests running on Travis at the same time as tests running on Jenkins), race conditions occur where the database is emptied while another test is using it, or the database is modified by both tests simultaneously, causing both of them to fail. Currently, this is mitigated by re-running the jobs until no collision happens.

Future iterations could address this issue by potentially using a local database instead of the same remote database or by implementing DB-locking on the server side by making each connection to the test db (and all its transactions) an atomic operation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.