Intermittent slow Redis operations and JournalStreamer write timeouts in Kubernetes HA setup on Dragonfly v1.36.0

## Summary

We are investigating intermittent latency spikes and possible gateway timeouts in a production Kubernetes environment using Dragonfly as a Redis-compatible distributed cache.

Application traces show requests periodically spending several seconds in Dragonfly/Redis cache operations, especially `set to cache level` and sometimes `get from cache level`. Around the same windows, Dragonfly master and replica logs show replication/socket errors such as `JournalStreamer write operation timeout`, `Input/output error`, replica disconnects, and partial re-syncs.

We are trying to determine whether this is a known issue, configuration problem, network sensitivity, or something fixed in a newer Dragonfly version.

## Environment

```text
Dragonfly image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.36.0
Deployment type: Kubernetes StatefulSet
Topology: HA master/replica setup
Service type: ClusterIP
Application client: StackExchange.Redis via .NET application
Cache library: FusionCache
Redis port: 6379
Dragonfly admin/replication port: 9999
```

Dragonfly container args:

```text
--alsologtostderr
--break_replication_on_master_restart=true
--primary_port_http_enabled=false
--admin_port=9999
--admin_nopass
--dir=s3:///backups
--snapshot_cron=0 0 * * *
```

Relevant environment/config:

```text
DFLY_maxmemory=10Gi
DFLY_s3_endpoint=
AWS_S3_FORCE_PATH_STYLE=true
AWS_REGION=
DFLY_requirepass=
```

Pod/resource status at inspection time:

```text
Master pod restart count: 0
Master pod status: Running
Master pod Ready: True
Kubernetes events on pod: none
CPU at inspection: low
Memory at inspection: below configured maxmemory
Service endpoint: single master endpoint only
```

Service routing appears correct. The application service points only to the Dragonfly master endpoint, selected by `role=master`.

## Observed application behavior

Application traces show slow cache-level operations. Examples:

```text
Trace example A:
HTTP request duration: ~7.9s
set to cache level: ~3.9s
set to cache level: ~4.0s
Factory/other work: not the bottleneck
HTTP status: 200
```

```text
Trace example B:
HTTP request duration: ~21.25s
get from cache level: ~10.25s
set to cache level / EXPIRE: ~10.98s
StackExchange.Redis timeout: timeout is 10000ms
Redis command: EXPIRE
StackExchange.Redis diagnostic: qs=306
HTTP status in trace: 200, but request duration is high enough to plausibly cause gateway/client timeout depending on ingress timeout
```

We also queried traces for slow cache-level operations:

```traceql
{ name =~ "get from cache level|set to cache level" && duration > 1s }
```

This returned multiple traces in the last 7 days, including requests lasting approximately:

```text
~6s
~7.9s
~10s
~13s
~16s
~18s
~21s
~24s
```

Many of these traces are dominated by `set to cache level` or `get from cache level`.

## Dragonfly master logs

At a time that correlates with one incident window, the master logged:

```text
W20260616 13:57:21.066556 common.cc:346] ReportError: JournalStreamer write operation timeout
I20260616 13:57:21.067067 dflycmd.cc:747] Replication error: JournalStreamer write operation timeout
I20260616 13:57:21.067128 dflycmd.cc:118] Disconnecting from replica :6379
I20260616 13:57:21.571267 dflycmd.cc:757] Registered replica :6379
I20260616 13:57:23.000258 dflycmd.cc:430] Transitioned into stable sync with replica :6379

W20260616 13:57:52.999413 common.cc:346] ReportError: JournalStreamer write operation timeout
I20260616 13:57:52.999425 dflycmd.cc:747] Replication error: JournalStreamer write operation timeout
I20260616 13:57:52.999447 dflycmd.cc:118] Disconnecting from replica :6379
I20260616 13:57:53.504801 dflycmd.cc:757] Registered replica :6379
I20260616 13:57:53.513677 dflycmd.cc:430] Transitioned into stable sync with replica :6379
```

Similar `JournalStreamer write operation timeout` / replica disconnect / re-sync events were also observed later, for example around:

```text
2026-06-16 14:15 UTC
2026-06-16 16:34 UTC
2026-06-16 17:15 UTC
2026-06-16 17:20 UTC
```

## Dragonfly replica logs

The replica had matching socket/I/O errors against the master admin/replication endpoint:

```text
W20260616 13:57:21.067576 common.cc:346] ReportError: Input/output error
W20260616 13:57:21.067647 replica.cc:780] Replication error in phase STABLE_SYNC with :9999, error: Input/output error, socket state: State: ESTABLISHED
W20260616 13:57:21.068049 protocol_client.cc:254] Socket error: Input/output error in :9999, socket info: State: ESTABLISHED
I20260616 13:57:21.068575 replica.cc:817] Exit stable sync
W20260616 13:57:21.068584 replica.cc:302] Error stable sync with :9999 (phase: TCP_CONNECTING): generic:5 Input/output error
I20260616 13:57:22.994455 replica.cc:646] Started partial sync with :9999
I20260616 13:57:23.000633 replica.cc:678] partial sync finished in 1.43 s
I20260616 13:57:23.000653 replica.cc:792] Transitioned into stable sync
```

A second similar event occurred around `13:57:53 UTC`, followed by partial sync recovery.

## Other observed log line

We also saw this, but it seems less related:

```text
CLIENT maint_notifications on moving-endpoint-type internal-ip failed with reason: Unknown subcommand or wrong number of arguments for 'MAINT_NOTIFICATIONS'. Try CLIENT HELP.
```

This looks like a client compatibility/probing warning and does not appear to correlate as strongly with the slow cache operations.

## Application logs

Both application pods show repeated FusionCache warnings around survey-cache keys:

```text
FUSION [N=FusionCache ... K=quest-:survey]: a synthetic timeout occurred while calling the factory
FUSION [N=FusionCache ... K=quest-:survey]: FAIL-SAFE activated (from memory)
```

These warnings are present on multiple application pods, so the symptom is not isolated to a single app pod. However, the most severe traces show the actual request duration dominated by Dragonfly/Redis cache-level operations rather than the application factory or upstream API.

## What we ruled out so far

We checked the following and did not find obvious issues:

```text
Dragonfly master pod restart: no restart
Dragonfly master pod readiness: Ready=True
Kubernetes pod events: none
Service routing: service points only to master endpoint
CPU/memory pressure at inspection: not observed
Application issue isolated to one pod: no, symptoms appear on multiple app pods
```

## Questions

1. Are `JournalStreamer write operation timeout` and replica disconnect/re-sync events known in Dragonfly `v1.36.0`?
2. Can replication/admin-port instability on `9999` affect client Redis operations on `6379`, for example causing slow responses or StackExchange.Redis `qs` backlog?
3. Are there known fixes after `v1.36.0` related to replication stability, socket handling, or slow Redis command responses?
4. Would you recommend upgrading from `v1.36.0` to `v1.37+` or a newer stable version for this scenario?
5. Are there recommended configuration changes for Kubernetes HA deployments to avoid this behavior?
6. Are there metrics or debug commands we should collect next when this happens again?

## Expected behavior

Dragonfly cache `GET`/`SET`/`EXPIRE` operations should complete in milliseconds under normal load, and replication should stay in stable sync without recurring `JournalStreamer write operation timeout` events.

## Actual behavior

Application traces periodically show cache-level operations taking multiple seconds, sometimes near the 10s client timeout. Dragonfly logs show replication write/socket errors and replica disconnect/re-sync events around some of the same windows.

No source files or tests are named; start by reproducing the v1.36.0 Kubernetes HA setup and collecting the correlated master and replica logs, client traces, and metrics requested. Compare behavior with a newer Dragonfly version and check whether port 9999 replication events coincide with port 6379 latency. Done requires a confirmed cause or a documented upgrade or configuration recommendation.

Written by the indexing model from the issue text.