MagicStack / MagicStack/asyncpg
backpressure on LISTEN?
還沒有人認領這個 Issue。
- 主要語言
- Python
- 星號
- 8.1k
- 分支
- 468
- PR 合併指標
- 30 天內沒有已合併 PR
描述
- asyncpg version: 0.18.3
- PostgreSQL version: 9.5
- Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?: - Python version: 3.6
- Platform:
- Do you use pgbouncer?:
- Did you install asyncpg with pip?:
- If you built asyncpg locally, which version of Cython did you use?:
- Can the issue be reproduced under both asyncio and
uvloop?:
My application subscribes to postgres notifications and fans them out to subscribers using streaming http.
I see that notifications emerge in connection._process_notification, where they are dispatched via call_soon(), each time calling _call_listener(), which will synchronously call the callback that I specify in conn.add_listener(). In the callback, I insert a task into the queue for asynchronous processing...
I see a lot of tasks inserted this way, e.g. ~5000 before my application has a chance to process them. Therefore application memory has big unpredictable spikes, like +300k
Question: how do you apply back-pressure in this setup? is there a way to limit the number of postgres notifications that are being converted to tasks, to control overall number of in-flight tasks? Thanks!
貢獻指南
這個儲存庫沒有索引到貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
研究方向
先閱讀 connection._process_notification、call_soon()、_call_listener() 和 conn.add_listener(),然後追蹤應用程式的佇列如何建立工作。該 issue 未指定目標檔案、測試或驗收行為;請確定結果是否應該是 API 層級對處理中的通知設定限制,並為該行為新增涵蓋測試。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- postgresql, python
- 領域
- databases
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 25/100