[Bug] Very short retry timeout on 'Subgraph failed with non-deterministic error'

El informe menciona SubgraphInstanceManager, NotificationListener, EthereumPollingBlockIngestor y errores de failover de PostgreSQL HA; comienza rastreando el manejo de reintentos para el fallo no determinista reportado. Se considera terminado cuando la ventana de reintento se puede configurar o, de otro modo, dura durante el failover descrito, con el comportamiento verificado frente a los errores reportados.

Bug report

Hello!

subgraph version: 0.35.1

I see some of our subgraphs stop with the following error logs. The crux is that it stops retrying after 6 seconds!

As I understand, it is caused by HA Postgres doing a leader change or going through a fail-over.

Is there a way to make subgraph to retry for a larger time span? I have not found a setting for that. I think it should try for several hours.

Thank you!

Relevant log output

Oct 14 02:31:02.795 ERRO Subgraph failed with non-deterministic error: Failed to transact block operations: store error: cannot execute UPDATE in a read-only transaction, retry_delay_s: 135, attempt: 0, sgd: 39, subgraph_id: QmRLj4v8iRGBYfRaGPan45QzehUQxQzwrKBU514nEqNZSA, component: SubgraphInstanceManager
Oct 14 02:30:51.304 ERRO Failed to connect notification listener: error connecting to server: failed to lookup address information: Name or service not known, retry_delay_s: 8, attempt: 3, channel: store_events, component: NotificationListener
Oct 14 02:30:51.278 ERRO Failed to connect notification listener: error connecting to server: failed to lookup address information: Name or service not known, retry_delay_s: 8, attempt: 3, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:51.086 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:50.341 ERRO Vacuum of subgraphs.subgraph_deployment failed: store error: cannot execute VACUUM during recovery, job: Vacuum subgraphs.subgraph_deployment, component: JobRunner
Oct 14 02:30:49.979 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:48.875 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:47.768 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:47.289 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 4, attempt: 2, channel: store_events, component: NotificationListener
Oct 14 02:30:47.263 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 4, attempt: 2, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:46.651 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:45.524 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:45.228 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 2, attempt: 1, channel: store_events, component: NotificationListener
Oct 14 02:30:45.199 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 2, attempt: 1, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:44.407 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:44.392 ERRO Subgraph instance failed to run: Failed to set subgraph status to `failed`: store error: cannot execute INSERT in a read-only transaction, sgd: 1, subgraph_id: QmP16jdXsZHFHgaxqUAVoTzHb5dhpfY83Tx2WfU3tvh3uo, component: SubgraphInstanceManager

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux