Node server_id is regenerated on every restart, and no storage tier is both persistent and per-node

Problem Statement

A dotCMS node's server_id is a freshly generated UUID on every JVM start. It is not persisted, not read from configuration, and has no environment override.

// com/dotcms/cluster/business/ServerAPI.java:15
final static Lazy<String> SERVER_ID = Lazy.of(UUIDUtil::uuid);

Every restart therefore makes a node a new cluster member to every server_id-keyed mechanism.

History — likely incidental, but please confirm

Persistence existed and was removed in 7189abfb60 — "Issue 31249 update license information (#31261)", 7 Feb 2025, the BSL relicensing change. It previously wrote the id to disk and read it back on later boots:

-    private File serverIdFile() {
-        String realPath = ConfigUtils.getDynamicContentPath() + File.separator + "license"
-                            + File.separator + "server_id.dat";
-    private void writeServerIdToDisk(String serverId) throws IOException { ... }
-        try (BufferedReader br = Files.newBufferedReader(serverFile.toPath())) {
-            SERVER_ID = br.readLine();

That PR's stated purpose was relicensing, so this looks incidental — but that is an inference, not a fact. Note the old path was under .../license/, so it may have been entangled with enterprise-licensed code that had to move. Please confirm intent with the author of #31261 before any fix.

Restoring the old logic would not fix dotCMS Cloud

This was the assumption in the first version of this issue, and reviewing representative Cloud StatefulSet manifests disproves it. The storage topology offers no location that is both persistent and per-pod:

Path	Persistent across restart?	Per-pod?
/data/shared	✅ backed by a ReadWriteMany PVC	❌ shared by every pod in the StatefulSet
/data/local/dotsecure (where getDynamicContentPath() resolves)	❌ ephemeral container filesystem	✅

The manifests define no volumeClaimTemplates, so there is no per-pod persistent volume.

Consequently:

• The old code's path was the ephemeral one, so simply reverting it changes nothing in Cloud.
• Moving the file to /data/shared would be actively worse: every pod would read the same id, and a multi-node cluster would believe it was a single node.

Any fix has to work without assuming per-node durable storage.

This is a solved problem elsewhere

Durable node identity is well-trodden ground, and existing systems cluster into four approaches:

Approach	Examples	Requires
Orchestrator/operator supplies the identity	Kafka broker.id, KRaft node.id, ZooKeeper myid	configuration only
Identity stored in a per-node data directory	Elasticsearch node ID, Consul node-id, Cassandra host ID	per-node persistent volume
Derived from stable host attributes	Consul from /etc/machine-id	a stable host
No durable identity; dynamic membership	Hazelcast, JGroups	the protocol tolerates it

dotCMS currently behaves like the fourth while having the needs of the second — per-node cursors, per-server heartbeat directories, own-event filtering. That mismatch is the defect.

Suggested direction — a precedence chain, not one mechanism

No single source works for both Cloud and on-premise, so resolve in order and log which tier was used:

1. Explicit configuration — e.g. DOT_SERVER_ID. Works in every topology; highest precedence.
2. Derived from orchestrator-supplied identity — pod name or hostname. StatefulSet pod names are stable across restarts by contract, so this needs no storage at all. Suggest a deterministic name-based UUID, e.g. UUID.nameUUIDFromBytes((clusterId + ":" + podName).getBytes()), so the value is stable, fits varchar(36), and does not collide across clusters.
3. Persisted file in a per-node durable path — for on-premise deployments that do have local persistent storage.
4. Random UUID with a warning — today's behaviour, as an explicit last resort rather than the silent default.

This avoids imposing a shared-volume requirement on on-premise operators while making Cloud deterministic. In Cloud the pod name is already available via the downward API, so tier 2 is a manifest addition rather than an architectural change.

Acceptance Criteria

• Intent of the removal in #31261 confirmed with its author, and recorded here.
• A node's server_id is stable across a restart in: Cloud (StatefulSet), on-premise with persistent storage, and local Docker.
• The resolution tier actually used is logged at startup, so an operator can tell which mechanism is in play.
• Falling back to a random UUID logs a warning naming the consequences, rather than proceeding silently.
• Automated test: identity is stable across a simulated restart for at least the configuration and derived-identity tiers.
• Per-server directories under assets/server/ stop accumulating one per boot.

Impact

Anything keyed on server_id treats a restarted node as a stranger:

• Cluster membership — a row per boot rather than per node.
• Heartbeats and per-server directories under assets/server/ accumulate without bound.
• Own-event filtering in SystemEventsJobDelegate (SERVER_ID.equals(event.getServerId())) stops recognising a node's own pre-restart events.
• The per-node delivery cursor from #36827 cannot be found after a restart, so the node seeds afresh and events committed during its downtime are lost.

It may also be a contributing cause of the loss reported in #36827 in environments where nodes restart, independently of the cursor defect described there.

dotCMS Version

Present on main. Introduced in 7189abfb60 (Feb 2025), so every release since.

Severity

High - Major functionality broken

Links

• Introducing commit 7189abfb60 / PR #31261 / issue #31249
• Found while verifying #36827 (PR #37288); restart recovery and the node-down scenario there cannot be tested until this is resolved

Start by reading com/dotcms/cluster/business/ServerAPI.java and confirm the removal intent with the author of PR #31261. Trace SystemEventsJobDelegate and the per-server directory and cursor behavior, then design and test stable configuration and derived identities across simulated restarts. Done means the selected tier is logged, random fallback warns, and server directories no longer accumulate.