MagicStack / MagicStack/asyncpg
backpressure on LISTEN?
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 8.1k
- 派生
- 468
- PR 合并指标
- 30 天内没有已合并 PR
描述
- asyncpg version: 0.18.3
- PostgreSQL version: 9.5
- Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?: - Python version: 3.6
- Platform:
- Do you use pgbouncer?:
- Did you install asyncpg with pip?:
- If you built asyncpg locally, which version of Cython did you use?:
- Can the issue be reproduced under both asyncio and
uvloop?:
My application subscribes to postgres notifications and fans them out to subscribers using streaming http.
I see that notifications emerge in connection._process_notification, where they are dispatched via call_soon(), each time calling _call_listener(), which will synchronously call the callback that I specify in conn.add_listener(). In the callback, I insert a task into the queue for asynchronous processing...
I see a lot of tasks inserted this way, e.g. ~5000 before my application has a chance to process them. Therefore application memory has big unpredictable spikes, like +300k
Question: how do you apply back-pressure in this setup? is there a way to limit the number of postgres notifications that are being converted to tasks, to control overall number of in-flight tasks? Thanks!
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
先阅读 connection._process_notification、call_soon()、_call_listener() 和 conn.add_listener(),然后跟踪应用程序的队列如何创建任务。该 issue 未指定目标文件、测试或验收行为;请确定结果是否应当是 API 层面对进行中的通知设置限制,并为该行为添加覆盖测试。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- postgresql, python
- 领域
- databases
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100