neume-network / neume-network/core

We should run a mini crawl frequently

Open
#67 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
27
Forks
11
PR merge metrics
No merged PRs in 30d

Description

  • neume-network/data runs a full crawl in roughly 2-3hrs every day
  • But to get e.g. feedback whether some strategies are properly integrated with each other can hence take a very long time. E.g. if we ship a change in strategy X, and downstream strategies break because of it, it may take up to one day for this to surface in a crash of the neume-network/data crawl
  • However, we don't have to crawl the entire chain and all platforms to get the information whether e.g. all strategies are properly integrated with each other. Instead, we could e.g. set up web3subgraph or any other starting strategy to e.g. only return a single id e.g. [{"id":"0x01ab7d30525e4f3010af27a003180463a6c811a6/1","createdAtBlockNumber":"14323199","platform":"sound"} and then do the crawl for just that id. (I've just taken the first value of: https://raw.githubusercontent.com/neume-network/data/main/results/web3subgraph-extraction)
  • If we found a set of start ids that resulted in all strategies being invoked at least once, we could probably finish an entire mini crawl in a matter of seconds or minutes: and we'd have a fast response/feedback for making coding mistakes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the full crawl in neume-network/data and the example value from results/web3subgraph-extraction. Identify a representative set of starting IDs that invokes every strategy, then determine how a mini crawl could run frequently. Done means the reduced crawl completes quickly and provides feedback when strategy integrations break.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data-engineering, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.