feast-dev / feast-dev/feast

feast serve segfaults with PostgreSQL online store on Linux/macOS

Open
#6,254 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
3d 16h
Merged PRs (30d)
15

Description

## What happens

`feast serve` with a PostgreSQL online store segfaults on the first request that hits `get_online_features`. Gunicorn restarts the worker, next request crashes it again. The server never serves a response.

SQLite and Elasticsearch online stores work fine under the same setup.

## Why

`import feast` pulls in pyarrow and protobuf's C/upb runtime. Both create C-level background threads at import time. Gunicorn then calls `fork()` to create workers. POSIX says only the calling thread survives a fork, so the pyarrow/protobuf threads are gone, but their internal C state (locked mutexes, thread handles, pointers) remains in the child's memory.

When the worker handles a request that goes through psycopg, libpq (psycopg's C backend) runs into this orphaned state and the process dies with signal 11 (SIGSEGV).

SQLite doesn't crash because it uses Python's built-in sqlite3 module. Elasticsearch doesn't crash because it uses a pure-Python HTTP client. Neither touches C-level state that was left behind by the dead threads.

Creating a fresh FeatureStore inside the worker's `load()` method doesn't help. The orphaned C state comes from `import feast` in the parent process, not from FeatureStore instantiation. The modules are already in `sys.modules` and their C extensions can't be reinitialized.

## Affected platforms

Any platform where `feast serve` uses gunicorn (Linux, macOS, all POSIX systems). Windows is not affected because it uses uvicorn.

## Reproducer

Minimal script, no server needed:

```python
import os
from feast import FeatureStore

store = FeatureStore(repo_path="my_pg_repo")
pid = os.fork()
if pid == 0:
store.get_online_features(
features=["driver_stats:conv_rate"],
entity_rows={"driver_id": [1001]},
)
os._exit(0)
else:
_, status = os.waitpid(pid, 0)
if os.WIFSIGNALED(status):
print(f"Killed by signal {os.WTERMSIG(status)}") # signal 11
```

Or through `feast serve`:

```bash
feast serve # with postgres online store
curl localhost:6566/get-online-features -H 'Content-Type: application/json' \
-d '{"features":["driver_stats:conv_rate"],"entities":{"driver_id":[1001]}}'
# worker dies with signal 11, respawns, dies again
```

Sample feature_store.yaml:

```yaml
project: test
registry: data/registry.db
provider: local
online_store:
type: postgres
host: localhost
port: 5432
database: feast
user: feast
password: feast
```

## Environment

- macOS / Linux
- Python 3.12
- psycopg 3.x (binary)
- Feast main branch

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.