`docker service logs $service` hangs intermittently

Description

We faced an issue that docker service logs $service hangs once in a while (sometimes on 5 or 6 run sometimes right away).

Reproduce

1. Setup swarm (in our case 4 nodes)
2. Setup service that runs for couple of minutes once a day (replicas=1) and prints some logs (not a lot maybe 100-200 lines)
3. Run the service (docker service update $SERVICE_NAME replicas=1 detach=false)
4. docker service logs $SERVICE_NAME Ctrl-c
5. docker service logs $SERVICE_NAME Ctrl-c
6. docker service logs $SERVICE_NAME Ctrl-c
7. repeat until it hangs

Expected behavior

docker service logs $SERVICE_NAME should never hang

docker version

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ced0996
 Built:             Thu Aug 31 00:00:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       a61e2b4
  Built:            Thu Aug 31 00:00:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:        0cae528dd6cb557f7201036e9f43420650207b58
 runc:
  Version:          1.1.7
  GitCommit:        f19387a6bec4944c770f7668ab51c4348d9c2f38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 27
  Running: 11
  Paused: 0
  Stopped: 16
 Images: 29
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: xzhnfsssg81j3mdzztervwpba
  Is Manager: true
  ClusterID: w5y4ld99b0ye372fdocfvl6k6
  Managers: 4
  Nodes: 4
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: REDACTED
  Manager Addresses:
   REDACTED:2377
   REDACTED:2377
   REDACTED:2377
   REDACTED:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.55-75.123.amzn2023.x86_64
 Operating System: Amazon Linux 2023
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 30.64GiB
 Name: REDACTED
 ID: d3bf143a-8f76-4b91-9b3e-384c6a3b3437
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

$ cat /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "100"
  }
}

Start by reproducing the hang with a four-node swarm, a short-lived one-replica service using json-file logging, and repeated docker service logs $SERVICE_NAME runs interrupted with Ctrl-C. Trace the docker service logs CLI entry point and its swarm log retrieval until the intermittent block is understood; done means repeated runs no longer hang and the behavior is covered by a regression test.