dotCMS / dotCMS/core

622 integration tests are in no MainSuite and never run in CI

Open
#37,254 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team : Maintenance Type : CI/CD
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

622 integration @Test methods across 99 test classes are not referenced by any MainSuite, so they never run in CI.

Integration tests only execute when their class is listed in one of dotcms-integration/src/test/java/com/dotcms/MainSuite{1a,1b,2a,2b,3a}.java. Registration is manual and nothing verifies it. A developer can write a test class, run it locally with -Dit.test=, watch it pass, open a PR — and CI will run none of it. Nothing about the build output distinguishes that from a test that ran and passed.

This was found while fixing #36827: six new integration classes had been written, run repeatedly, and reported green, and every one of them was unregistered. Checking whether the problem was wider produced the numbers below.

Count
Integration *Test classes under dotcms-integration/src/test/java 665
Not referenced by any MainSuite 109
…excluding abstract/base helper classes, with real @Test methods 99
Orphaned @Test methods 622
⚠️ Some of these are probably excluded on purpose — check before adding

Do not bulk-add these to suites. A class may be absent from every suite for perfectly good reasons, and adding it blindly could break the build or resurrect work that was deliberately parked. Before touching any entry, establish which of these it is:

  • Deliberately excluded — quarantined as flaky, too slow for the suite budget, or superseded by another test.
  • Work in progress — belongs to a feature still in flight, where the test is expected to fail or the feature is not finished.
  • Environment-dependent — needs something CI does not provide.
  • Genuinely orphaned — simply never registered, which is the case this issue is really about.

Only the last category should be added. For the others the correct outcome is to record why, so the next person auditing this does not have to re-derive the reasoning — an @Ignore with a reason, or a comment in the suite, is enough.

The list below is therefore an inventory to triage, not a work list to burn down.

Steps to reproduce the inventory
cat dotcms-integration/src/test/java/com/dotcms/MainSuite*.java > /tmp/allsuites.txt
find dotcms-integration/src/test/java -name "*Test.java" | sed 's#.*/##; s/\.java$//' | sort -u |
while read t; do
  grep -q "\b$t\b" /tmp/allsuites.txt || echo "$t"
done
Acceptance Criteria
  • Every one of the 99 unregistered classes is triaged into one of: deliberately excluded, work in progress, environment-dependent, or genuinely orphaned.
  • Genuinely orphaned classes are added to an appropriate suite (respecting the "avoid adding more to MainSuite2a" note in that file's header).
  • Classes that stay out have the reason recorded in the code — @Ignore("reason") or a comment — so the exclusion is visible rather than implied by absence.
  • A CI guard fails the build when an integration test class containing @Test methods is in no suite and carries no recorded exclusion. Without this, the inventory drifts again — this is the part that stops recurrence.
  • The guard is proven to work: adding an unregistered test class makes CI fail.
  • The 7 unregistered runonce migration tests are triaged as part of this (listed below) — migration tests are cheap and there is rarely a reason to exclude them.
Additional Context

Unregistered runonce migration tests (small, self-contained, likely genuine orphans):

  • Task05300UpdateIndexNameLengthTest
  • Task211020CreateHostIntegrityCheckerResultTablesTest
  • Task221018CreateVariantFieldInMultiTreeTest
  • Task230119MigrateContentToProperPersonaTagAndRemoveDupTagsTest
  • Task241014AddTemplateValueOnContentletIndexTest
  • Task250910AddAnalyticsDashboardPortletToMenuTest
  • Task251029RemoveContentTypesLegacyPortletFromLayoutsTest

Related but separate gap: Task260507CreateS3VanityAliasTable has no test at all — not an unregistered test, a missing one. Worth covering while migration test coverage is being looked at, but it is a different problem from this issue.

Priority rationale. Filed as Medium rather than High: nothing is broken for users today, and no product functionality is impaired. The risk is that regressions in these areas land undetected, and that engineers reasonably believe coverage exists where it does not. Triage may well raise it.

Full inventory — 99 unregistered classes with `@Test` counts (click to expand)
Class @Test methods
ContentletIndexAPIImplMigrationIntegrationTest 31
PostgresJobQueueIntegrationTest 27
OSIndexAPIImplIntegrationTest 26
ContentImportResourceIntegrationTest 25
HostWebAPIImplIntegrationTest 24
ContentFactoryIndexOperationsOSIntegrationTest 22
ImportContentletsProcessorIntegrationTest 18
OSClientConfigTest 16
OSSiteSearchAPIIntegrationTest 15
VersionedIndicesAPITest 14
OSSearchAPIImplIntegrationTest 14
ContentletIndexOperationsOSIntegrationTest 14
AppsUtilTest 14
FilesCollectorIntegrationTest 13
SamlWebUtilsTest 12
PageResourceHelperTest 12
BulkRefreshResourceIntegrationTest 12
UtilHTMLTest 11
OSClientProviderIntegrationTest 10
MimeTypeUtilsTest 10
JobQueueManagerAPIIntegrationTest 10
HostUtilTest 10
VisitorAPITest 9
VariantWebAPIImplIntegrationTest 9
UsageResourceIntegrationTest 9
MultiTreeCacheTest 9
AnalyticsAPIImplTest 9
UserPersistenceTest 8
JobQueueHelperIntegrationTest 8
AccessTokenRenewJobTest 8
S3VanityStaticPublishingIntegrationTest 7
OSMappingAPIImplIntegrationTest 7
LuceneQueryBuilderTest 7
CurrentVariantWebInterceptorTest 7
ContentletIndexAPIImplPhaseSwitchIntegrationTest 7
TempFileAPITest 6
ReindexQueueFactoryTest 6
PopulateIdentifierBaseTypeJobTest 6
PermissionBitFactoryImplGetPermittedIdsTest 6
PermissionBitAPIImplFilterCollectionTest 6
OSCreateContentIndexIntegrationTest 6
DBPropertiesDataSourceStrategyTest 6
Task251029RemoveContentTypesLegacyPortletFromLayoutsTest 5
DotWebdavHelperTest 5
VelocityLiveModeTest 4
UniqueFieldsValidationInitializerTest 4
PushedAssetsFactoryTest 4
PopulateIdentifierBaseTypeUtilTest 4
OsBootstrapForbiddenIndexTest 4
ExperimentsCacheTest 4
ApiTokenResourceTest 4
WorkflowHistoryTest 3
RoleRevocationPermissionCacheTest 3
ReindexQueueAPITest 3
LanguageVariableFactoryTest 3
JobQueueManagerAPICDITest 3
FieldAPIImplIntegrationTest 3
DetailPageTransformerImplTest 3
Task250910AddAnalyticsDashboardPortletToMenuTest 2
Task221018CreateVariantFieldInMultiTreeTest 2
SpeedyAssetServletTest 2
SamlWebInterceptorTest 2
PageViewSerializerTest 2
JobProcessorDiscoveryTest 2
FieldVariableLazyHydrationTest 2
ExperimentResultsQueryFactoryIntegrationTest 2
EditLanguageActionTest 2
DotJSONCacheTest 2
DotFolderTransformerTest 2
DockerSecretDataSourceStrategyTest 2
CubeJSClientFactoryIntegrationTest 2
CompanyUtilTest 2
BulkEmbeddingsRunnerTest 2
BinaryBlockFilJoinerImplTest 2
WorkflowProcessorTest 1
UserTest 1
TomcatDatasourceStrategyTest 1
Task241014AddTemplateValueOnContentletIndexTest 1
Task230119MigrateContentToProperPersonaTagAndRemoveDupTagsTest 1
Task211020CreateHostIntegrityCheckerResultTablesTest 1
Task05300UpdateIndexNameLengthTest 1
SystemEnvDataSourceStrategyTest 1
SecurityKeyResetTest 1
ResetPermissionsJobTest 1
ResetApproversActionletTest 1
LanguageVariablesHelperTest 1
HTMLPageAssetAPIImplTest 1
FieldTest 1
ExperimentFactoryIntegrationTest 1
DotObjectMapperProviderTest 1
CustomAttributeCacheImplTest 1
CountExperimentsWithURLParameterGoalMetricTypeTest 1
CountExperimentsWithReachPageGoalMetricTypeTest 1
CountExperimentsWithExitRateGoalMetricTypeTest 1
CountExperimentsWithBounceRateGoalMetricTypeTest 1
CountExperimentsEditedInThePast30DaysMetricTypeTest 1
CompanyAPITest 1
CheckinContentActionletTest 1
BundlerUtilIntegrationTest 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the MainSuite1a, MainSuite1b, MainSuite2a, MainSuite2b, and MainSuite3a files under dotcms-integration/src/test/java/com/dotcms, then run the inventory commands in the issue. Done means all 99 classes are triaged, genuine orphans are registered, exclusions have recorded reasons, and a CI guard is proven to fail for an unregistered test class.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.