[Bug] High subgraph query throughput causes excessive traffic on the database primary node

未关闭
#5,336 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
停滞
技术栈
rust, sql
领域
backend, databases

调研方向

未指定源文件、测试或入口点。首先追踪传入子图查询的数据库选择,以及在报告吞吐量下列出的区块、健康检查、错误和部署 SQL。完成的标准是:查询流量被适当地引导离开主数据库,并且根据预期的索引行为验证了任何缓存行为。

由索引模型根据 Issue 内容生成。

描述

bug
Bug report

Hello,

We have deployed TheGraph internally and are using it as the blockchain data source for some of our services. It works great, thanks a lot for building such a great product! Lately, we've been scaling up our infrastructure to answer more queries. A part of this effort included adding a read replica to lower the load on our primary database instance (the goal is to have all query nodes get data from the read replicas, so that the primary instance can focus on indexing tasks). Configuration went pretty smoothly after reading the docs and we are indeed seeing some traffic targeting our read replicas. However, we're still seeing surprisingly high usage on our primary database instance, and this usage increases as we get more queries (while we only expect read replicas to be impacted since this is querying data that does not need to perform any writes).

Some analysis of the issue with Performance Insights reveals an elevated rate of queries on the primary instance that seem to correlate with incoming queries given the high throughput. We think these should be targeted to read replicas to scale up properly. Here are some queries along with their rate and aggregated CPU usage metric (10% means the primary instance spends 10% of all its CPU time processing queries like this).

CPU % Rate (calls/s) SQL
42 268.64 SELECT "chain1"."blocks"."number", coalesce(data->'block'->>'timestamp', data->>'timestamp') FROM "chain1"."blocks" WHERE "chain1"."blocks"."hash" = $1 LIMIT $2
11 1348.71 SELECT 1
6 268.64 SELECT EXISTS (SELECT "subgraphs"."subgraph_error"."vid", "subgraphs"."subgraph_error"."id", "subgraphs"."subgraph_error"."subgraph_id", "subgraphs"."subgraph_error"."message", "subgraphs"."subgraph_error"."block_hash", "subgraphs"."subgraph_error"."handler", "subgraphs"."subgraph_error"."deterministic", "subgraphs"."subgraph_error"."block_range" FROM "subgraphs"."subgraph_error" WHERE "subgraphs"."subgraph_error"."subgraph_id" = $1 AND "subgraphs"."subgraph_error"."deterministic" AND block_range @> $2)
4 537.27 SELECT "subgraphs"."subgraph_deployment"."deployment", "subgraphs"."subgraph_deployment"."reorg_count", "subgraphs"."subgraph_deployment"."max_reorg_depth", "subgraphs"."subgraph_deployment"."latest_ethereum_block_number", "subgraphs"."subgraph_deployment"."latest_ethereum_block_hash", "subgraphs"."subgraph_deployment"."earliest_block_number" FROM "subgraphs"."subgraph_deployment" WHERE "subgraphs"."subgraph_deployment"."deployment" = $1 LIMIT $2
2 268.65 SELECT "subgraphs"."subgraph_version"."deployment" FROM ("subgraphs"."subgraph_version" INNER JOIN "subgraphs"."subgraph" ON "subgraphs"."subgraph"."current_version" = "subgraphs"."subgraph_version"."id") WHERE "subgraphs"."subgraph"."name" = $1 LIMIT $2

For reference, at the time of the measurement, we were processing approximately 270 reqs/s.

Besides directing these queries to the database read replicas, caching some of their results for a few seconds seems sensible as well and should not have a noticeable impact on the behaviour of thegraph.

Thanks in advance for your help

Relevant log output

No response

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

Deployed version: 0.33.0
Linux

主要语言
Rust
星标
3.2k
派生
1.1k
平均合并
4 天 1 小时
30 天内合并 PR
1

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

graphprotocol/graph-node 的其他 Issue

查看 graphprotocol/graph-node 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。