apache / apache/pulsar-client-python

[Bug] Python pulsar-client producer does not produce messages with gunicorn `--preload` flag

未關閉
#52 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
75
分支
53
PR 合併指標
30 天內沒有已合併 PR

描述

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar brokers version 2.9
python 3.9
Tested on macOS and in production (Linux)

### Minimal reproduce step

### 1. Python env setup

Create a venv with the following packages installed:
`requirements.txt`
```
Flask==2.0.1
gevent==21.8.0
gunicorn==20.1.0
pulsar-client==2.9.2
```

### 2. Pulsar setup

Start a standalone pulsar broker compatible with a 2.9 client
```
~/dev/pulsar
> bin/pulsar standalone -nss
```

### 3. Python script
`test_pulsar.py`
```python
import logging
import uuid
from datetime import datetime

from _pulsar import Result
from flask import Flask, make_response
from pulsar import Client, MessageId, schema

logging.basicConfig(level=logging.INFO, format='[%(asctime)s] [%(levelname)s] [%(name)s] %(message)s')
logger = logging.getLogger(__name__)
logger.info('Loading API')

app = Flask(__name__)

client = Client('pulsar://127.0.0.1:6650', authentication=None)

producer = client.create_producer(
'non-persistent://public/default/test-gunicorn',
producer_name=f'my-producer-{uuid.uuid4()}',
schema=schema.StringSchema()
)

def callback(res: Result, _msg_id: MessageId):
logger.info(f'Callback result here! Event acknowledged by the broker.')

@app.post('/post-message')
def post_pulsar_message():
logger.info('Calling producer.send_async now, in the next lines there should be the callback result')
dt = datetime.now()
producer.send_async(content=f'dt={dt.isoformat()}', callback=callback)
logger.info('After producer.send_async, returning the http response')
return '', 201

@app.get('/')
def healthcheck():
logger.info('API Running fine')
return make_response({'status': 'healthy'})

logger.info('API started')
```

### 4. Run configurations

In your virtual environment, run any of the following:

**valid configuration**
```
gunicorn test_pulsar:app --workers=2
```

**broken configuration**
```
gunicorn test_pulsar:app --workers=2 --preload
```

### 5. Call the producing endpoint
```
curl -X POST http://127.0.0.1:8000/post-message
```

### What did you expect to see?

In the **valid configuration** , once the endpoint is called, the message is correctly produced and we can see the callback log line.

```
[INFO] [test_pulsar] Calling producer.send_async now, in the next lines there should be the callback result
[INFO] [test_pulsar] After producer.send_async, returning the http response
[INFO] [test_pulsar] Callback result here! Event acknowledged by the broker.
```

### What did you see instead?

In the **invalid configuration** the producer line is executed but no message is produced

```
[INFO] [test_pulsar] Calling producer.send_async now, in the next lines there should be the callback result
[INFO] [test_pulsar] After producer.send_async, returning the http response

```

### Anything else?

I tried also with the `producer.send` version, but it blocks and times out.

I also made sure that the monkey-patching expected in gunicorn runtimes does not affect the issue (tried with and without).

It seems to be an issue with the execution context of the actual producer, being incompatible with a gunicorn runtime, at least with those params.

This is an issue because using the `--preload` param is a must-have in production, as it generates only one of each static value (producer, db pool, caches and so on) instead of one per worker, and not having it is expensive in the compute / memory / operations side.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

貢獻指南

開啟貢獻指南

研究方向

從 test_pulsar.py 中的重現開始,並將帶有和不帶有 --preload 的兩個 Gunicorn 命令與獨立 Pulsar broker 進行比較。該 issue 未指定原始檔或測試;完成標準是使用 --preload 執行時,POST endpoint 產生一則訊息並記錄 callback 日誌。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
flask, python
領域
distributed-systems
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。