[Bug] Very short retry timeout on 'Subgraph failed with non-deterministic error'

The report names SubgraphInstanceManager, NotificationListener, EthereumPollingBlockIngestor, and PostgreSQL HA failover errors; begin by tracing the retry handling for the reported non-deterministic failure. Done means the retry window can be configured or otherwise lasts through the described failover, with behavior verified against the reported errors.

Bug report

Hello!

subgraph version: 0.35.1

I see some of our subgraphs stop with the following error logs. The crux is that it stops retrying after 6 seconds!

As I understand, it is caused by HA Postgres doing a leader change or going through a fail-over.

Is there a way to make subgraph to retry for a larger time span? I have not found a setting for that. I think it should try for several hours.

Thank you!

Relevant log output

Oct 14 02:31:02.795 ERRO Subgraph failed with non-deterministic error: Failed to transact block operations: store error: cannot execute UPDATE in a read-only transaction, retry_delay_s: 135, attempt: 0, sgd: 39, subgraph_id: QmRLj4v8iRGBYfRaGPan45QzehUQxQzwrKBU514nEqNZSA, component: SubgraphInstanceManager
Oct 14 02:30:51.304 ERRO Failed to connect notification listener: error connecting to server: failed to lookup address information: Name or service not known, retry_delay_s: 8, attempt: 3, channel: store_events, component: NotificationListener
Oct 14 02:30:51.278 ERRO Failed to connect notification listener: error connecting to server: failed to lookup address information: Name or service not known, retry_delay_s: 8, attempt: 3, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:51.086 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:50.341 ERRO Vacuum of subgraphs.subgraph_deployment failed: store error: cannot execute VACUUM during recovery, job: Vacuum subgraphs.subgraph_deployment, component: JobRunner
Oct 14 02:30:49.979 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:48.875 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:47.768 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:47.289 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 4, attempt: 2, channel: store_events, component: NotificationListener
Oct 14 02:30:47.263 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 4, attempt: 2, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:46.651 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:45.524 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:45.228 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 2, attempt: 1, channel: store_events, component: NotificationListener
Oct 14 02:30:45.199 ERRO Failed to connect notification listener: db error: ERROR: cannot execute LISTEN during recovery, retry_delay_s: 2, attempt: 1, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Oct 14 02:30:44.407 ERRO Trying again after block polling failed: Ingestor error: store error: cannot execute INSERT in a read-only transaction, provider: mainnet, component: EthereumPollingBlockIngestor
Oct 14 02:30:44.392 ERRO Subgraph instance failed to run: Failed to set subgraph status to `failed`: store error: cannot execute INSERT in a read-only transaction, sgd: 1, subgraph_id: QmP16jdXsZHFHgaxqUAVoTzHb5dhpfY83Tx2WfU3tvh3uo, component: SubgraphInstanceManager

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux