Coordinated identity/template/vault change_mode

Nomad 1.7 allows for multiple identity{} blocks for creating JWTs for use with third party services. Expiration is a core concept in JWTs as it is in any modern credential system, and so it is assumed every identity{} will require a ttl and rotation. Traditionally tasks use the change_mode parameter to determine how Nomad informs them about template and token changes. #17805 is an example of how our existing parameters are insufficient to provide a good user experience around template changes.

The core problem is that informing tasks of changes is rarely free or even fast. A full task restart may take many seconds or more before the task is ready to start accepting requests and performing its work again. Since all instances of the task will have the same TTLs, they will all be restarting around the same time. This has led to user outages if the splay or jitter isn't sufficient to spread out restarts/reloading.

Multiple identity blocks will make this worse than the existing template/vault problems as they add more TTLs that could cause correlated restarts.

Intra-task Coordination

Since the Nomad Client performs the change_mode action, it can coalesce multiple reloads into a single change_mode invocation. This should work particularly well with identities where Nomad controls the rate of change, and identities could be eagerly reloaded long before their TTL if it decreases the total number of restarts necessary. For example:

If you have a task with 3 identities:

1. AWS TTL=3h
2. UserAPI TTL=2h
3. Mesh TTL=10h

The min(ttl) is 2h, so we should renew all JWTs every 2 hours (since creating JWTs is cheap).

However what if they have differing change_modes? If UserAPI is noop and AWS is restart, we would need to restart every 3 hours even if the last renewal was 1 hour prior. If a template causes a restart between the 2h token renewal and the 3h restart, there's no need to still do the restart at 3h.

Nomad Clients should track and coordinate all change_mode invocations across template/vault/identity blocks to ensure minimum disruption to tasks.

Inter-task Coordination

Minimizing disruptions within a task does not to prevent correlated restarts between instances of the task spread out over the cluster. There's currently no mechanism by which Nomad Clients can coordinate with one another directly, and since the Vault and template changes are external to Nomad entirely, Nomad Servers cannot perform change coordination on behalf of distributed Clients.

Idea 1: Deterministic Jitter

Using NOMAD_ALLOC_INDEX to deterministically choose an amount to jitter the change mode invocation should spread those invocations out across Clients. However this requires having some notion of "restart time" carefully aligning TTLs + count + restart_time to ensure all instances can be changed in time.

Idea 2: Locks

Nomad has locks now! Each task could attempt to acquire a lock based on the name of the component being restarted (and job and task group) to serialize restarts across the cluster.

While this does incur more server overhead, and wouldn't work for disconnected clients, it has the massive benefit of giving users an escape hatch: an operator could manually acquire the lock to stop a change from rolling out. This could be used to prevent a bad template change from causing an outage, but may be too low-level/difficult to reasonably use for that purpose.

Workaround: Noop

One approach users have taken with template{} that can also work with Vault and identity is to use change_mode="noop" and use application specific logic to opportunistically reload templates/tokens off disk.

This can be accomplished a variety of ways:

• Apps can periodically reload tokens. Reloading at <1/2 the min(ttl) of all templates and tokens should avoid using expired tokens.
• Apps can reload templates/tokens on request failures. If a Vault call fails: reload the Vault token and try again. If a call using Workload Identity fails: reload the JWT and try again. etc. While this is a good fallback mechanism that should probably be widely implemented, there are a lot of implementation complexities and third party apps can't be expected to behave this way.
• A single template { change_mode = "restart" } can be used to watch a key in Nomad or Consul that is used by an external service to signal when a reload should happen. This requires an external app that can coordinate these reloads which is only really realistic for coordinating template reloads as the external app won't know when a Vault token or JWT needs reloading.

Workaround: SIGHUP

Using SIGHUP (or any non-terminating signal) makes a barrage of changes feasible:

identity {
  name = "aws"
  aud  = ["s3"]
  ttl  = "4h"

  change_mode   = "signal"
  change_signal = "SIGHUP"
}

While it would be tempting to treat this as the recommended solution, the usability of signals is poor: they are not cross-platform and have a lot of sharp edges for app developers to work through.

This approach, while using a fairly standard UNIX idiom, still relies on the app behaving in a specific way. For tasks running third party apps, they may not have the ability to define this behavior.

The issue names no files, tests, or implementation entry points. Start by reviewing how Nomad clients currently handle change_mode across template, Vault, and identity blocks, then determine which coordination design is acceptable. Done is not defined yet; the issue needs a scoped design and maintainer decision before implementation.