622 integration tests are in no MainSuite and never run in CI
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
622 integration @Test methods across 99 test classes are not referenced by any MainSuite, so they never run in CI.
Integration tests only execute when their class is listed in one of dotcms-integration/src/test/java/com/dotcms/MainSuite{1a,1b,2a,2b,3a}.java. Registration is manual and nothing verifies it. A developer can write a test class, run it locally with -Dit.test=, watch it pass, open a PR — and CI will run none of it. Nothing about the build output distinguishes that from a test that ran and passed.
This was found while fixing #36827: six new integration classes had been written, run repeatedly, and reported green, and every one of them was unregistered. Checking whether the problem was wider produced the numbers below.
| Count | |
|---|---|
Integration *Test classes under dotcms-integration/src/test/java |
665 |
Not referenced by any MainSuite |
109 |
…excluding abstract/base helper classes, with real @Test methods |
99 |
Orphaned @Test methods |
622 |
⚠️ Some of these are probably excluded on purpose — check before adding
Do not bulk-add these to suites. A class may be absent from every suite for perfectly good reasons, and adding it blindly could break the build or resurrect work that was deliberately parked. Before touching any entry, establish which of these it is:
- Deliberately excluded — quarantined as flaky, too slow for the suite budget, or superseded by another test.
- Work in progress — belongs to a feature still in flight, where the test is expected to fail or the feature is not finished.
- Environment-dependent — needs something CI does not provide.
- Genuinely orphaned — simply never registered, which is the case this issue is really about.
Only the last category should be added. For the others the correct outcome is to record why, so the next person auditing this does not have to re-derive the reasoning — an @Ignore with a reason, or a comment in the suite, is enough.
The list below is therefore an inventory to triage, not a work list to burn down.
Steps to reproduce the inventory
cat dotcms-integration/src/test/java/com/dotcms/MainSuite*.java > /tmp/allsuites.txt
find dotcms-integration/src/test/java -name "*Test.java" | sed 's#.*/##; s/\.java$//' | sort -u |
while read t; do
grep -q "\b$t\b" /tmp/allsuites.txt || echo "$t"
done
Acceptance Criteria
- Every one of the 99 unregistered classes is triaged into one of: deliberately excluded, work in progress, environment-dependent, or genuinely orphaned.
- Genuinely orphaned classes are added to an appropriate suite (respecting the "avoid adding more to
MainSuite2a" note in that file's header). - Classes that stay out have the reason recorded in the code —
@Ignore("reason")or a comment — so the exclusion is visible rather than implied by absence. - A CI guard fails the build when an integration test class containing
@Testmethods is in no suite and carries no recorded exclusion. Without this, the inventory drifts again — this is the part that stops recurrence. - The guard is proven to work: adding an unregistered test class makes CI fail.
- The 7 unregistered runonce migration tests are triaged as part of this (listed below) — migration tests are cheap and there is rarely a reason to exclude them.
Additional Context
Unregistered runonce migration tests (small, self-contained, likely genuine orphans):
Task05300UpdateIndexNameLengthTestTask211020CreateHostIntegrityCheckerResultTablesTestTask221018CreateVariantFieldInMultiTreeTestTask230119MigrateContentToProperPersonaTagAndRemoveDupTagsTestTask241014AddTemplateValueOnContentletIndexTestTask250910AddAnalyticsDashboardPortletToMenuTestTask251029RemoveContentTypesLegacyPortletFromLayoutsTest
Related but separate gap: Task260507CreateS3VanityAliasTable has no test at all — not an unregistered test, a missing one. Worth covering while migration test coverage is being looked at, but it is a different problem from this issue.
Priority rationale. Filed as Medium rather than High: nothing is broken for users today, and no product functionality is impaired. The risk is that regressions in these areas land undetected, and that engineers reasonably believe coverage exists where it does not. Triage may well raise it.
Full inventory — 99 unregistered classes with `@Test` counts (click to expand)
| Class | @Test methods |
|---|---|
ContentletIndexAPIImplMigrationIntegrationTest |
31 |
PostgresJobQueueIntegrationTest |
27 |
OSIndexAPIImplIntegrationTest |
26 |
ContentImportResourceIntegrationTest |
25 |
HostWebAPIImplIntegrationTest |
24 |
ContentFactoryIndexOperationsOSIntegrationTest |
22 |
ImportContentletsProcessorIntegrationTest |
18 |
OSClientConfigTest |
16 |
OSSiteSearchAPIIntegrationTest |
15 |
VersionedIndicesAPITest |
14 |
OSSearchAPIImplIntegrationTest |
14 |
ContentletIndexOperationsOSIntegrationTest |
14 |
AppsUtilTest |
14 |
FilesCollectorIntegrationTest |
13 |
SamlWebUtilsTest |
12 |
PageResourceHelperTest |
12 |
BulkRefreshResourceIntegrationTest |
12 |
UtilHTMLTest |
11 |
OSClientProviderIntegrationTest |
10 |
MimeTypeUtilsTest |
10 |
JobQueueManagerAPIIntegrationTest |
10 |
HostUtilTest |
10 |
VisitorAPITest |
9 |
VariantWebAPIImplIntegrationTest |
9 |
UsageResourceIntegrationTest |
9 |
MultiTreeCacheTest |
9 |
AnalyticsAPIImplTest |
9 |
UserPersistenceTest |
8 |
JobQueueHelperIntegrationTest |
8 |
AccessTokenRenewJobTest |
8 |
S3VanityStaticPublishingIntegrationTest |
7 |
OSMappingAPIImplIntegrationTest |
7 |
LuceneQueryBuilderTest |
7 |
CurrentVariantWebInterceptorTest |
7 |
ContentletIndexAPIImplPhaseSwitchIntegrationTest |
7 |
TempFileAPITest |
6 |
ReindexQueueFactoryTest |
6 |
PopulateIdentifierBaseTypeJobTest |
6 |
PermissionBitFactoryImplGetPermittedIdsTest |
6 |
PermissionBitAPIImplFilterCollectionTest |
6 |
OSCreateContentIndexIntegrationTest |
6 |
DBPropertiesDataSourceStrategyTest |
6 |
Task251029RemoveContentTypesLegacyPortletFromLayoutsTest |
5 |
DotWebdavHelperTest |
5 |
VelocityLiveModeTest |
4 |
UniqueFieldsValidationInitializerTest |
4 |
PushedAssetsFactoryTest |
4 |
PopulateIdentifierBaseTypeUtilTest |
4 |
OsBootstrapForbiddenIndexTest |
4 |
ExperimentsCacheTest |
4 |
ApiTokenResourceTest |
4 |
WorkflowHistoryTest |
3 |
RoleRevocationPermissionCacheTest |
3 |
ReindexQueueAPITest |
3 |
LanguageVariableFactoryTest |
3 |
JobQueueManagerAPICDITest |
3 |
FieldAPIImplIntegrationTest |
3 |
DetailPageTransformerImplTest |
3 |
Task250910AddAnalyticsDashboardPortletToMenuTest |
2 |
Task221018CreateVariantFieldInMultiTreeTest |
2 |
SpeedyAssetServletTest |
2 |
SamlWebInterceptorTest |
2 |
PageViewSerializerTest |
2 |
JobProcessorDiscoveryTest |
2 |
FieldVariableLazyHydrationTest |
2 |
ExperimentResultsQueryFactoryIntegrationTest |
2 |
EditLanguageActionTest |
2 |
DotJSONCacheTest |
2 |
DotFolderTransformerTest |
2 |
DockerSecretDataSourceStrategyTest |
2 |
CubeJSClientFactoryIntegrationTest |
2 |
CompanyUtilTest |
2 |
BulkEmbeddingsRunnerTest |
2 |
BinaryBlockFilJoinerImplTest |
2 |
WorkflowProcessorTest |
1 |
UserTest |
1 |
TomcatDatasourceStrategyTest |
1 |
Task241014AddTemplateValueOnContentletIndexTest |
1 |
Task230119MigrateContentToProperPersonaTagAndRemoveDupTagsTest |
1 |
Task211020CreateHostIntegrityCheckerResultTablesTest |
1 |
Task05300UpdateIndexNameLengthTest |
1 |
SystemEnvDataSourceStrategyTest |
1 |
SecurityKeyResetTest |
1 |
ResetPermissionsJobTest |
1 |
ResetApproversActionletTest |
1 |
LanguageVariablesHelperTest |
1 |
HTMLPageAssetAPIImplTest |
1 |
FieldTest |
1 |
ExperimentFactoryIntegrationTest |
1 |
DotObjectMapperProviderTest |
1 |
CustomAttributeCacheImplTest |
1 |
CountExperimentsWithURLParameterGoalMetricTypeTest |
1 |
CountExperimentsWithReachPageGoalMetricTypeTest |
1 |
CountExperimentsWithExitRateGoalMetricTypeTest |
1 |
CountExperimentsWithBounceRateGoalMetricTypeTest |
1 |
CountExperimentsEditedInThePast30DaysMetricTypeTest |
1 |
CompanyAPITest |
1 |
CheckinContentActionletTest |
1 |
BundlerUtilIntegrationTest |
1 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the MainSuite1a, MainSuite1b, MainSuite2a, MainSuite2b, and MainSuite3a files under dotcms-integration/src/test/java/com/dotcms, then run the inventory commands in the issue. Done means all 99 classes are triaged, genuine orphans are registered, exclusions have recorded reasons, and a CI guard is proven to fail for an unregistered test class.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100