duneanalytics / duneanalytics/spellbook

[BUG] solana bot trades lineage is unable to run full refresh and data is stale

Open
#7,989 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug dbt: solana
Dominant language
Python
Stars
1.5k
Forks
1.4k
Avg merge
3d 16h
Merged PRs (30d)
22

Description

all models which feed into dex_solana.bot_trades are out of date in terms of propagating bug fixes / new DEXs added upstream to dex_solana.trades. when changes are made, or new DEX added, to dex_solana.trades, the table rebuilds fully in prod efficiently. once complete, all bot trades models kick off since they read from DEX. the bot trades models will not run in prod, regardless of compute infrastructure in place (number of worker nodes in cluster, amount of concurrency, etc).

due to this, typically we won't run these models after dex is refreshed. we also have not added a new bot trades model since.

in order to ensure data quality & add more models, we need to revamp the design for which they are built in order to run efficiently in prod.

the most impactful learning recently for spellbook performance on the trino engine has been to materialize early, and materialize often. we can use staging tables frequently to get data in the shape we need to use downstream in finals tables. trino performance is nuked when you inject a CTE or subquery into a join condition. the planner doesn't handle it well. if you materialize that CTE or subquery, then use as a table join in downstream model, it performs much better.

i think bonkbot is one of the worst performance models, it would be good to open a fresh PR with that one to rearchitect to follow these patterns.

for any of these upstream models built as staging tables, avoid displaying on data explorer with post hooks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the models feeding dex_solana.bot_trades from dex_solana.trades, then inspect the bonkbot model and its staging-table configuration. Compare a full refresh in production with the current behavior; done means bot trades models refresh successfully, propagate upstream changes, and avoid exposing staging tables in Data Explorer.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.