[CI] Distributed tests are too slow.
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
There are currently many issues regarding to the unit tests in `tests/distributed/test_distributed_sampling.py`:
* They are very slow. A successful run takes about 15 mins.
* They often timeout (40 mins) when there are multiple PRs being tested. See an example here: https://dgl-jenkins-eksvpc-2136217999.us-west-2.elb.amazonaws.com/blue/organizations/jenkins/dgl/detail/master/997/pipeline/448/
* They sometimes hang indefinitely and will eventually timeout.
The root cause of these issues is that each testcase will conduct server creation, client-server connection, perform actual test (e.g., sampling or other DistGraph queries) and resource cleanup. The design leads to the following problem:
* Launching servers/clients using Python multiprocessing package and making connections are very expensive. These works are usually redundant since most of the testcase are to test the DistGraph query operations not the resource initialization and teardown.
* Frequently launching and tearing down servers/clients often lead to issues like port contention or busy OS scheduling. This is also not the typical use case of DistDGL where users often create servers/clients once and train for many iterations.
My suggestion is to change the way of writing unittests for the distributed package, breaking it into three phases:
1. *Resource initialization*: This includes launching server processes, loading data from disk, creating DistGraph object, and connecting clients to servers. This phase should be decoupled from the number of test cases.
2. *Actual test*: Each test case should not repetitively initialize resources. Instead, they submit requests and verify the correctness of the response.
3. *Resource teardown*: Close all the created resources after tests are finished.
The new way corresponds very well with pytest's fixture design: https://docs.pytest.org/en/6.2.x/fixture.html#what-fixtures-are . At a high-level, we should:
* Implement different `DistGraph` or server configurations as fixtures.
* Test cases are functions that depend on the initialized fixtures.
Pytest will also handle the lifetime of fixtures to ensure resources are properly released after the depending tests are finished.
@Rhett-Ying @frozenbugs
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.