feast serve segfaults with PostgreSQL online store on Linux/macOS
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 15
Description
## What happens
`feast serve` with a PostgreSQL online store segfaults on the first request that hits `get_online_features`. Gunicorn restarts the worker, next request crashes it again. The server never serves a response.
SQLite and Elasticsearch online stores work fine under the same setup.
## Why
`import feast` pulls in pyarrow and protobuf's C/upb runtime. Both create C-level background threads at import time. Gunicorn then calls `fork()` to create workers. POSIX says only the calling thread survives a fork, so the pyarrow/protobuf threads are gone, but their internal C state (locked mutexes, thread handles, pointers) remains in the child's memory.
When the worker handles a request that goes through psycopg, libpq (psycopg's C backend) runs into this orphaned state and the process dies with signal 11 (SIGSEGV).
SQLite doesn't crash because it uses Python's built-in sqlite3 module. Elasticsearch doesn't crash because it uses a pure-Python HTTP client. Neither touches C-level state that was left behind by the dead threads.
Creating a fresh FeatureStore inside the worker's `load()` method doesn't help. The orphaned C state comes from `import feast` in the parent process, not from FeatureStore instantiation. The modules are already in `sys.modules` and their C extensions can't be reinitialized.
## Affected platforms
Any platform where `feast serve` uses gunicorn (Linux, macOS, all POSIX systems). Windows is not affected because it uses uvicorn.
## Reproducer
Minimal script, no server needed:
```python
import os
from feast import FeatureStore
store = FeatureStore(repo_path="my_pg_repo")
pid = os.fork()
if pid == 0:
store.get_online_features(
features=["driver_stats:conv_rate"],
entity_rows={"driver_id": [1001]},
)
os._exit(0)
else:
_, status = os.waitpid(pid, 0)
if os.WIFSIGNALED(status):
print(f"Killed by signal {os.WTERMSIG(status)}") # signal 11
```
Or through `feast serve`:
```bash
feast serve # with postgres online store
curl localhost:6566/get-online-features -H 'Content-Type: application/json' \
-d '{"features":["driver_stats:conv_rate"],"entities":{"driver_id":[1001]}}'
# worker dies with signal 11, respawns, dies again
```
Sample feature_store.yaml:
```yaml
project: test
registry: data/registry.db
provider: local
online_store:
type: postgres
host: localhost
port: 5432
database: feast
user: feast
password: feast
```
## Environment
- macOS / Linux
- Python 3.12
- psycopg 3.x (binary)
- Feast main branch
Contributor guide
Assessment
This issue has not been assessed yet.