eclipse-platform / eclipse-platform/eclipse.platform

Reactivate tests: disabled tests collection

Open
#525 2 comments 0 reactions 0 assignees View on GitHub
good first issue test
Dominant language
Java
Stars
165
Forks
174
Avg merge
2d 8h
Merged PRs (30d)
22

Description

### Current Situation
Several tests have been deactivated over the years for different reasons, such as temporary incompatibilities with a specific platform/OS, bugs in the code, indeterministic behavior of the test etc.

These disablements are done by adding some prefix to `test...` methods in JUnit 3, by removing or commenting out `@Test` in JUnit 4 and 5, or by using `@Ignore` or `@Disabled`. In addition, several tests are only deactivated for specific platforms using early returns or assume statements within the test methods.

### Goal
Since in most cases there is no additional, unique documentation or tagging of these disablements, they are not easy to find. This issue is supposed to be an umbrella to document disabled tests (existing ones and potentially further ones).

The purpose of this issue is to collect the disabled tests to have a central overview and provide the chance to reactivate and, if necessary, fix them. It also serves as a "good first issue" (see https://github.com/eclipse-platform/eclipse.platform.text/pull/212#issuecomment-1594576748) for interested new contributors, which can take any of the collected tests and try to fix them to learn about the Eclipse platform code while, at the same time, already having the chance to make valuable contributions to the project.

### Process
I see the following ideas as good starting points for identifying disabled tests:
- First concentrate on a single project, such as `org.eclipse.core.tests.resources`
- Use different techniques to find disabled tests:
1. Completely disabled tests:
- Search for missing or commented out `@Test` annotation (JUnit 4 and 5)
- Search for tests not starting with `test` (JUnit 3), in particular those starting with `_test`
2. Tests disabled for specific platforms:
- Search for accesses to OS constants in `Platform`, such as `OS_WIN32` and validate if they are used to disable tests for specific platforms
- Search for accesses to methods `isLinux()` etc. in `ResourceTest` subclasses and validate if they are used to disable tests for specific platforms
- Search for `assumeFalse` or `assumeTrue` statements that make tests only run on specific platforms

#### Examples
- The tests modified in https://github.com/eclipse-platform/eclipse.platform.text/pull/212
- Tests in `org.eclipse.core.tests.resources.regression.Bug_032076`
- The `skipTest` helper in `org.eclipse.core.tests.resources.usecase.SnapshotTest`

### Disabled in Maven
Some tests are disabled in Maven builds, such that integration builds do not reveal whether they fail or not.
- [ ] The bundle [`org.eclipse.core.tests.net`](https://github.com/eclipse-platform/eclipse.platform/blob/cebe637f94a94480c00b24f29d460ec163ba973c/team/tests/org.eclipse.core.tests.net/pom.xml#L28) is completely disabled. Tests run fine withing Eclipse on Windows, but Tycho execution fails, probably because of missing platform-dependent fragments (maybe have a look at how SWT deals with that).
- [ ] `org.eclipse.team.tests.core.AllTeamUITests` and `org.eclipse.team.tests.ui.synchronize.AllTeamSynchronizeTests` are not executed as the test suite [only executes `org.eclipse.team.tests.core.AllTeamTests`](https://github.com/eclipse-platform/eclipse.platform/blob/cebe637f94a94480c00b24f29d460ec163ba973c/team/tests/org.eclipse.team.tests.core/pom.xml#L26). Tests run fine within Eclipse on WIndows, but Tycho fails to execute some of the `ScopeTests` and the `AllTeamSynchronizeTest`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.