OpenFn / OpenFn/apollo

Apollo raises 429 errors when refreshing the adaptor list

Open
#576 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Jupyter Notebook
Stars
5
Forks
10
Avg merge
2d 20h
Merged PRs (30d)
17

Description

For example:

https://openfn.sentry.io/issues/7599689182/?environment=production&project=4509824591069184&query=is%3Aunresolved%20429&referrer=issue-stream

The error occurs because we are hitting rate limits calling GitHub's servers to pull down adaptor metadata.

Solutions:

  1. Use a GitHub API token to increase the request limit
  2. Use another source

As an aside, in the current implementation if an adaptor receives a 429, it;ll be IGNORED by apollo like it doesn't exist. We are going to drop in a fix for this so that the error are less harmful .

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Apollo's adaptor metadata refresh path and trace the calls to GitHub's servers. Reproduce or inspect the 429 handling, then verify that rate-limited adaptors are not silently treated as missing and that the selected rate-limit mitigation works.

Written by the indexing model from the issue text.

Assessment

Tech stack
github
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.