Snapshot can be unavailable for seconds for embedded replica client

Syncing
2024-04-11T19:12:29Z app[48e5369b732e98] iad [info]2024-04-11T19:12:29.416879Z ERROR tower_http::trace::on_failure: response failed classification=Code: 14 latency=10 ms
2024-04-11T19:12:29Z app[48e5369b732e98] iad [info]2024-04-11T19:12:29.417006Z ERROR consumer: Error syncing: Replication(Client(Status { code: Unavailable, message: "snapshot not found", metadata: MetadataMap { headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "access-control-expose-headers": "grpc-status,grpc-message,grpc-status-details-bin", "content-type": "application/grpc-web+proto", "date": "Thu, 11 Apr 2024 19:12:28 GMT", "fly-request-id": "01HV796RPY4984ZQ3RGC4ZMPBR-iad", "via": "2 fly.io, 1.1 fly.io", "content-length": "0", "server": "Fly/e7839a863 (2024-04-09)", "vary": "origin", "vary": "access-control-request-method", "vary": "access-control-request-headers"} }, source: None }))
2024-04-11T19:12:29Z app[48e5369b732e98] iad [info]2024-04-11T19:12:29.894703Z  INFO consumer: Syncing
2024-04-11T19:12:29Z app[48e5369b732e98] iad [info]2024-04-11T19:12:29.909297Z ERROR tower_http::trace::on_failure: response failed classification=Code: 14 latency=4 ms
2024-04-11T19:12:29Z app[48e5369b732e98] iad [info]2024-04-11T19:12:29.909421Z ERROR consumer: Error syncing: Replication(Client(Status { code: Unavailable, message: "snapshot not found", metadata: MetadataMap { headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "access-control-expose-headers": "grpc-status,grpc-message,grpc-status-details-bin", "content-type": "application/grpc-web+proto", "date": "Thu, 11 Apr 2024 19:12:28 GMT", "fly-request-id": "01HV796S6GX5Y2FTAEJAB66ZT5-iad", "via": "2 fly.io, 1.1 fly.io", "content-length": "0", "server": "Fly/e7839a863 (2024-04-09)", "vary": "origin", "vary": "access-control-request-method", "vary": "access-control-request-headers"} }, source: None }))
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.395192Z  INFO consumer: Syncing
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.409973Z ERROR tower_http::trace::on_failure: response failed classification=Code: 14 latency=4 ms
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.410083Z ERROR consumer: Error syncing: Replication(Client(Status { code: Unavailable, message: "snapshot not found", metadata: MetadataMap { headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "access-control-expose-headers": "grpc-status,grpc-message,grpc-status-details-bin", "content-type": "application/grpc-web+proto", "date": "Thu, 11 Apr 2024 19:12:30 GMT", "fly-request-id": "01HV796TNDKD48DBBSKXP50Y7J-iad", "via": "2 fly.io, 1.1 fly.io", "content-length": "0", "server": "Fly/e7839a863 (2024-04-09)", "vary": "origin", "vary": "access-control-request-method", "vary": "access-control-request-headers"} }, source: None }))
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.894765Z  INFO consumer: Syncing
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.909802Z ERROR tower_http::trace::on_failure: response failed classification=Code: 14 latency=4 ms
2024-04-11T19:12:31Z app[48e5369b732e98] iad [info]2024-04-11T19:12:31.909918Z ERROR consumer: Error syncing: Replication(Client(Status { code: Unavailable, message: "snapshot not found", metadata: MetadataMap { headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "access-control-expose-headers": "grpc-status,grpc-message,grpc-status-details-bin", "content-type": "application/grpc-web+proto", "date": "Thu, 11 Apr 2024 19:12:30 GMT", "fly-request-id": "01HV796V51HGAYGCG4H73YHGZJ-iad", "via": "2 fly.io, 1.1 fly.io", "content-length": "0", "server": "Fly/e7839a863 (2024-04-09)", "vary": "origin", "vary": "access-control-request-method", "vary": "access-control-request-headers"} }, source: None }))

The issue names no files, tests, or entry points. Start by tracing the embedded replica sync path that produces the logged snapshot not found error and reproduce the transient failure; done means the reported unavailability is addressed and the sync behavior is covered by an appropriate test.