fly-apps / fly-apps/keydb

Setup doesn't respect dynamic scaling

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
10
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Hi, after scaling down from 2 to 1, the cache has stopped working. This happens because we use `replica-serve-stale-data no`. This implies that a replica can't respond with data that wasn't synced with the master.

```
2021-08-29T18:13:46.994500165Z app[558aa91a] fra [info] Main child exited normally with code: 0
2021-08-29T18:13:46.995057953Z app[558aa91a] fra [info] Starting clean up.
2021-08-29T18:13:47.011565534Z app[558aa91a] fra [info] Umounting /dev/vdc from /data
2021-08-29T18:13:47.262972129Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:13:47.259 * Connecting to MASTER my-ip:2:6379
2021-08-29T18:13:47.265616064Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:13:47.263 * MASTER <-> REPLICA sync started
2021-08-29T18:14:48.460649059Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:14:48.458 # Timeout connecting to the MASTER...
2021-08-29T18:14:49.465835265Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:14:49.463 * Connecting to MASTER my-ip:2:6379
2021-08-29T18:14:49.468028292Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:14:49.466 * MASTER <-> REPLICA sync started
2021-08-29T18:15:50.673772013Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:15:50.672 # Timeout connecting to the MASTER...
2021-08-29T18:15:51.678936361Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:15:51.675 * Connecting to MASTER my-ip:2:6379
2021-08-29T18:15:51.681534926Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:15:51.679 * MASTER <-> REPLICA sync started
2021-08-29T18:16:52.869276300Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:16:52.866 # Timeout connecting to the MASTER...
2021-08-29T18:16:53.875176665Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:16:53.872 * Connecting to MASTER my-ip:2:6379
2021-08-29T18:16:53.877146484Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:16:53.875 * MASTER <-> REPLICA sync started
2021-08-29T18:17:20.937210786Z app[fd18d80c] fra [info] Reaped child process with pid: 28872, exit code: 0
2021-08-29T18:17:54.070869869Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:17:54.067 # Timeout connecting to the MASTER...
2021-08-29T18:17:55.077356332Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:17:55.074 * Connecting to MASTER my-ip:2:6379
2021-08-29T18:17:55.080193605Z app[fd18d80c] fra [info] keydb | 545:584:S 29 Aug 2021 18:17:55.077 * MASTER <-> REPLICA sync started
```

Client error: `MASTERDOWN Link with MASTER is down, and replica-serve-stale-data is set to 'no.'`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.