`DEADLINE_EXCEEDED: GetKeyValue() timed out`

In multi-node jobs onn clusters like Alps, Leonardo and Perlmutter we occasionally see failures like (this is using 8 processes on 8 nodes, for a total of 32 GPUs)
```
E0000 00:00:1743713172.513857 698835 pjrt_stream_executor_client.cc:3072] Execution of replica 0 failed: DEADLINE_EXCEEDED: GetKeyValue() timed out with key: devices=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29]; stream=0; groups=[[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29]]; root_device=0; num_local_participants=2 and duration: 10m [type.googleapis.com/tensorflow.CoordinationServiceError='']
```