dotCMS / dotCMS/core

Make the Docker-backed test recipes runnable from a git worktree with deterministic, non-colliding ports

Open
#37,221 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Build Team : Falcon Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

None of the Docker-backed test recipes can be used from a git worktree today. Every one of them
pins host ports and container names to fixed values, so a second checkout — or a running
just dev-run — collides with it. Developers working one worktree per issue currently cannot run
these suites at all without stopping whatever else is up.

just test-postman Experiments_Resource     # tomcat 8080, wiremock 50505, debug 5005
just test-integration -Dit.test=MyTest     # db 5437, es 9207, os-upgrade 9201
just test-karate                           # tomcat 8080, management 8090
just test-e2e                              # tomcat 8080

Why it matters. Measured on PR run 32875684455: from push to knowing whether the experiment
Postman group passed is 50 minutes (the job itself is 17 min; the rest is the upstream build
phase), and a backend PR fans out 11 Postman jobs alongside the integration and Karate
batteries. That is a fine merge gate and a terrible authoring loop. Writing or debugging a single
collection or a single integration test locally should be minutes, and today it is blocked outright
inside a worktree.

What is pinned, per recipe

Recipe Tomcat DB OpenSearch Other fixed
test-postman 8080 (dotcms-postman/pom.xml:35) dynamic dynamic wiremock 50505, JDWP 5005 (via -Pdebug, always passed)
test-integration dynamic 5437 (dotcms-integration/pom.xml:65) 9207 (:66) os-upgrade 9201 (:67)
test-karate 8080 (test-karate/pom.xml:24) dynamic dynamic management 8090 (:25)
test-e2e 8080 (core-web/apps/dotcms-ui-e2e/pom.xml:41) dynamic dynamic
dev-run dynamic dynamic dynamic — (already correct; dotCMS/pom.xml:44 sets fabric8.dynamic-port=true and leaves db.port/es.port commented out)

Root cause — one mechanism, four local patches

docker.run.context is ${context.parent}_${context.module}_${context.name}
(parent/pom.xml:194), and container names follow ${docker.run.context}_%a (:199). Because
context.module is the artifactId, the modules do not collide with each other — the collision is
always the same module run from two different checkouts, because context.name resolves to
ext.default.context.name = default (environments/environment.properties) and nothing ever
derives it. Volumes (data-local, data-shared, db, es) hang off the same context and clash the
same way.

The ports have no automatic derivation either. parent/pom.xml declares them in fabric8's
dynamic-port form (tomcat.port:8080, db.port:5432, es.port:9200, tomcat.ssl.port:8443,
management.port:8090), which assigns a free host port unless the property is set — and each
test module sets the ones it cares about, for its own historical reason. dev-run is the one place
that leaves them alone, and it is consequently the only recipe that already behaves.

So this is not a Postman problem. It is one missing derivation, patched around in four places.

Approach

  • Derive in one place. context.name and a port offset are computed once — in parent/pom.xml
    or a shared justfile helper — and every Docker-backed recipe consumes them. No per-module
    reimplementation.
  • Outside a worktree, nothing changes. The main checkout keeps every current value
    (8080/8443/8090/50505/5005, 5437/9207/9201) and the current container and volume
    names. The existing pins stay as the defaults.
  • Inside a worktree, detect it and derive deterministically. The recipe recognises a worktree
    (e.g. git rev-parse --git-common-dir differing from --git-dir) and derives the offset and
    context.name from a stable key — the worktree directory name or branch. Same worktree, same
    ports on every run, so a debugger attach or a bookmarked URL keeps working; different worktrees
    never overlap.
  • An explicitly passed port always wins, over both the derived worktree value and the default.
    At the Maven level this already works — a command-line -Dtomcat.port=9090 overrides the pinned
    property. What is missing is a passthrough in the just recipes, which today take only a
    collection or test name.

Deliberately out of scope: rewriting the Postman collections. dotcms-postman/pom.xml:34
carries the comment that some tests "currently fail unless docker external port is same as
internal"
, naming ApiToken_Resource — it holds a collection variable remoteTokenPort = 8080,
and it lives in the default group. Thirteen other collection files contain a literal :8080, but
most of those sit in descriptions or rawUrl fields rather than real requests, and
Experiments_Resource contains none. So this issue derives the ports and verifies which
collections actually break on a non-8080 host port; fixing any that do is a separate, scoped
follow-up. Until then -Dtomcat.port=8080 is the documented escape hatch.

Acceptance Criteria

Default path — no behavior change

  • Run from the main checkout (not a worktree), each of test-postman, test-integration,
    test-karate and test-e2e uses the same host ports, container names and volume names as
    today — no flags required, no observable difference.

Worktree path — deterministic isolation

  • The derivation of context.name and the port offset lives in one place and is consumed by
    every Docker-backed recipe. No module reimplements it.
  • Run from a git worktree, each of the four recipes completes without a port binding error and
    without reusing another checkout's containers or volumes.
  • Every host port a recipe binds is derived from the worktree's stable key — Tomcat, Tomcat SSL,
    management, WireMock, JDWP debug, Postgres, OpenSearch and os-upgrade. No fixed value survives
    into the worktree path.
  • The derived values are deterministic: the same worktree produces the same ports and the
    same context.name on every run.
  • Two different worktrees run the same recipe at the same time and both complete; neither
    reports a port conflict, a container-name clash, nor picks up the other's volumes.
  • A worktree recipe runs while just dev-run is up in the main checkout, and neither disturbs
    the other.
  • The resolved server URL and ports are printed at the start of the run, so a developer can
    attach a debugger or open the instance without reading the Maven log.

Escape hatches and failure modes

  • Each recipe accepts an explicit port — e.g. just test-postman Experiments_Resource 9090
    and that value wins over both the derived worktree value and the default.
  • Explicit Maven flags still win too: -Dtomcat.port, -Dcontext.name, -Ddebug.port,
    -Dwiremock.port, -Ddb.port, -Des.port on the command line override the derived values.
    (This already works at the Maven level today; the recipes must not break it.)
  • If a derived port is already taken, the run fails with a message naming the port and how to
    override it — not a raw Docker bind error.
  • just postman-stop and just test-integration-stop stop only the containers belonging to the
    current worktree, and cleanup-at-start does not remove another worktree's containers or
    volumes.

Cleanup surfaced while scoping this

  • test-karate/pom.xml:23 carries the ApiToken_Resource.postman_collection comment
    copy-pasted from the Postman module, but Karate runs no Postman collections. Remove or correct
    it.
  • core-web/apps/dotcms-ui-e2e/pom.xml declares <tomcat.port>8080</tomcat.port> twice in the
    same <properties> block (lines 41 and 52). Collapse to one.

Verification and follow-up

  • Verified end to end from a worktree for each recipe, including two worktrees running the same
    recipe concurrently, and from the main checkout with just dev-run up.
  • Each Postman collection group is run once on a derived (non-8080) port and the result
    recorded, so we know exactly which groups depend on host port == container port. Any that
    break get a separate follow-up issue rather than a fix here.
  • The behavior is documented where a developer will find it — the justfile recipe comments
    plus docs/testing/, including -Dtomcat.port=8080 as the escape hatch for the affected
    Postman groups.
Priority

Medium

Additional Context
  • -Pdebug is passed unconditionally by the test-postman recipe, so debug.port=5005 is always
    bound — a worktree run collides with any other debuggable dotCMS instance before Tomcat is even
    reached.
  • dev-run is the reference implementation: it leaves the fabric8 dynamic-port syntax alone and
    therefore already survives a second checkout. The goal is for the test recipes to reach the same
    place without losing determinism.
  • Related: #34722 (development environment setup — shared services, worktree integration).
  • environments/dev/user-dev.properties is gitignored and already supports per-developer overrides
    with the ext. prefix; it is a candidate location for a manual opt-out.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with parent/pom.xml and the justfile recipes for test-postman, test-integration, test-karate, and test-e2e; compare them with dotCMS/pom.xml and the module POM locations listed in the issue. Run the four recipes from the main checkout and a worktree to establish current ports and names. Done means deterministic isolated runs, explicit-port overrides, safe cleanup, printed endpoints, and documented behavior are verified across two concurrent worktrees.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, git, java
Domain
build-system, developer-experience, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.