cancervariants / cancervariants/variation-normalization

Postgres query errors for compose.yaml when under parallel load due to low `shm_size` default

Open
#627 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
15
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

The default docker (and podman) value for the `shm_size` config is 64MiB.

https://docs.docker.com/engine/containers/run/ (search `--shm-size`)

The default value for Postgres `shared_buffers` config is 128MiB. (though googling around it seems like other things can go into shared memory too like stuff for thread management/coordination)

When Postgres is handling concurrent UTA queries, it may try to allocate memory more memory than docker will allow and hit OS memory errors.

Error message example:
```
could not resize shared memory segment "/PostgreSQL.748496398" to 16777216 bytes: No space left on device
```

https://dba.stackexchange.com/questions/275378/dev-shm-size-recommendation-for-postgres-database-in-docker

### Steps to reproduce

I don't have a specific replication but I was hitting this nondeterministically with 8 threads processing variants sending some number, up to 8, of queries to the UTA postgres at the same time.

### Expected behavior

A reasonable number of threads should be able to call into liftover functions and query UTA simultaneously.

### Current behavior

Sometimes it doesn't work

### Possible reason(s)

_No response_

### Suggested fix

Increase `shm_size` on the UTA container in compose.yaml or add note

or tell postgres in the UTA container to disable concurrent queries, or reduce the postgres shared_buffers size

or mount a volume to /dev/shm in the UTA container

https://the-optimizer.medium.com/cracking-the-shared-buffers-how-dev-shm-impacts-postgresql-performance-on-kubernetes-8225fe4a6d59

### Branch, commit, and/or version

`0.15.0` (applies to all)

### Screenshots

_No response_

### Environment details

macos, podman, the version of UTA/postgres from compose.yaml @ 0.15.0

### Additional details

_No response_

### Contribution

None

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.