hasura / hasura/graphql-engine

Retry remote schema introspection when inconsistent

Open
#7,064 0 comments 8 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

We had some downtime overnight so as tech lead I'm trying to figure out the best architectural way to prevent this going forward.

What I think happened is our host Heroku patched our Postgres database by promoting an updated follower to primary, updating the database URL env variable, and restarting our apps (our backend app and Hasura instance are both hosted on Heroku). Hasura starts quicker than our app so I expect Hasura would've queried our remote schema on boot, got an error, and started reporting inconsistent metadata for the remote schema.

Thankfully according to [this comment](https://github.com/hasura/graphql-engine/issues/6727#issuecomment-828315912) as of v2 that doesn't bring down the entire Hasura instance anymore, just the remote schema. But large chunks of our app rely on our remote schema so that's still an issue for us.

---

I would like it if Hasura could retry introspecting remote schemas with an exponential backoff so that this issue would sort itself out. Do you think that's possible to implement please?

Otherwise we'll have to set up a cron job to reload metadata every 10 minutes just in case, which feels like a hacky workaround.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or code entry points are named. Start by investigating Hasura's remote schema introspection during startup and the reported Heroku/Postgres failover scenario; done would mean remote schemas retry introspection with exponential backoff and recover without a manual metadata reload.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, postgres
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.