stan-dev / stan-dev/cmdstanpy

Understanding `CmdStanModel` Behavior with `os.fork`

Open
#780 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
198
Forks
81
PR merge metrics
No merged PRs in 30d

Description

I've run into a somewhat odd (and likely edge-case) situation where CmdStanModel.sample behaves unexpectedly after a call to os.fork. I'm providing pseudocode now in the hopes that there is a fairly simple explanation, but happy to make a minimum working example if need be.

Say I run something structured like the following:

import os
from cmdstanpy import CmdStanModel

def run_job():
    # Compile the model
    mod = CmdStanModel(stanfile, cpp_options={"STAN_THREADS": True})
    
    # Run sampling on a detached child process
    pid = os.fork()
    if pid == 0:
        os.setsid()
        mod.sample(data = data, chains=4)
        os._exit()
    else:
        return pid

Even though my CPU has 32 cores, this will only run two chains at once--specifically, Chains 1 and 3 run, then Chains 2 and 4 run. If, however, I remove the cpp_options={"STAN_THREADS": True} kwarg, then everything behaves as expected: all chains sample in parallel.

Is there any immediate explanation for this discrepancy in behavior?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the interaction between CmdStanModel.sample, cpp_options={"STAN_THREADS": True}, and the os.fork path shown in the report. Reproduce the two cases with a minimal working example, then compare chain scheduling after fork; done means the discrepancy is explained and any necessary behavior or documentation change is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.