rethink latest_adaptors list

Apollo maintains a list of the latest adaptor versions, with a basic description of each. This used by the workflow generation service.

This list is saved as a JSON file and is refreshed every hour.

Building the list is quite expensive, with one call to the github API to fetch a list of adaptor names, and one call per adaptor latest version to fetch the package JSON.

This is very inefficient, not very kind to githubs servers, and has a tendency to throw 429 errors.

I'd like to rethink the implementation of this entirely. We need to consider the other services which depend on adaptor metadata - and all-in-one solution would be nice.

options

• The embed_docsite service has a better architecture: whenever the docsite is updated, a GH action calls up to apollo. Apollo then downloads the 5mb docs.json file and embeds the information within for RAG. Maybe we should update this to be more like: docs calls apollo, apollo downloads docs.json, apollo triggers docsite embeddings AND latest_adaptors update.
• Preserve the cache in the database, not a JSON file,so that it persists

Start by tracing how latest_adaptors is refreshed, where its JSON cache is stored, and which services consume adaptor metadata. Compare that flow with the embed_docsite architecture and evaluate the database-cache option; done requires a decided approach that addresses GitHub API load and the dependent services.