NVIDIA / NVIDIA/NeMo-Retriever

[BUG]: Issue running Python API quickstart on macOS

Open
#1,049 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Version

25.9.0

Which installation method(s) does this occur on?

No response

Describe the bug.

While trying to run the Python API quickstart example here https://docs.nvidia.com/nemo/retriever/latest/extraction/quickstart-library-mode/ on my local laptop (a macbook), I encountered this multiprocessing error due to a not fully initialized process attempting to spawn another process:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html
Full traceback: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jbourbeau/work/NVIDIA/nv-ingest/ingestion.py", line 7, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_runners import run_pipeline
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/pipeline_runners.py", line 25, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_builders import setup_ingestion_pipeline
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/pipeline_builders.py", line 16, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.stage_builders import (
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/stage_builders.py", line 18, in <module>
    from nv_ingest.framework.orchestration.ray.stages.extractors.chart_extractor import ChartExtractorStage
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/stages/extractors/chart_extractor.py", line 12, in <module>
    from nv_ingest_api.internal.extract.image.chart_extractor import extract_chart_data_from_image_internal
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/extract/image/chart_extractor.py", line 20, in <module>
    from nv_ingest_api.internal.primitives.nim.model_interface.ocr import OCRModelInterface
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/primitives/nim/model_interface/ocr.py", line 19, in <module>
    from nv_ingest_api.internal.primitives.nim.model_interface.decorators import (
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/primitives/nim/model_interface/decorators.py", line 13, in <module>
    manager = Manager()
              ^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/managers.py", line 562, in start
    self._process.start()
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

Traceback (most recent call last):
  File "/Users/jbourbeau/work/NVIDIA/nv-ingest/ingestion.py", line 7, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_runners import run_pipeline
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/pipeline_runners.py", line 25, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_builders import setup_ingestion_pipeline
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/pipeline_builders.py", line 16, in <module>
    from nv_ingest.framework.orchestration.ray.util.pipeline.stage_builders import (
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/util/pipeline/stage_builders.py", line 18, in <module>
    from nv_ingest.framework.orchestration.ray.stages.extractors.chart_extractor import ChartExtractorStage
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest/framework/orchestration/ray/stages/extractors/chart_extractor.py", line 12, in <module>
    from nv_ingest_api.internal.extract.image.chart_extractor import extract_chart_data_from_image_internal
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/extract/image/chart_extractor.py", line 20, in <module>
    from nv_ingest_api.internal.primitives.nim.model_interface.ocr import OCRModelInterface
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/primitives/nim/model_interface/ocr.py", line 19, in <module>
    from nv_ingest_api.internal.primitives.nim.model_interface.decorators import (
  File "/Users/jbourbeau/work/demos/nv-ingest/nvingest/lib/python3.12/site-packages/nv_ingest_api/internal/primitives/nim/model_interface/decorators.py", line 13, in <module>
    manager = Manager()
              ^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/managers.py", line 566, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/Users/jbourbeau/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/lib/python3.12/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError

This error is coming from the first import from nv_ingest in Step 2 here https://docs.nvidia.com/nemo/retriever/latest/extraction/quickstart-library-mode/#step-2-ingest-documents. That import itself

from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_runners import run_pipeline

is actually a MRE for me locally.

I think this has to do with the default multiprocessing start method on macOS being "spawn", as opposed to "fork" being the default on linux. If I manually set the start method to "fork" like this

import multiprocessing as mp
mp.set_start_method("fork")

from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_runners import run_pipeline

then the import works as expected locally for me.

A couple of additional points:

  • I didn't noticed until later that using linux is mentioned towards the top of the example (my bad for not catching that). I'm not sure if nv_ingest intends to support running on a mac or not. If it's not, it might be worth somehow highlighting that more prominently in the docs (or even checking the local system and raising if it's not linux).
  • I was surprised that the run_pipeline import caused multiple processes to be spawned. Usually imports don't have such large side effects (though, obviously, that's not a strict rule -- just a common convention)
Minimum reproducible example
from nv_ingest.framework.orchestration.ray.util.pipeline.pipeline_runners import run_pipeline
Relevant log output
Other/Misc.

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Step 2 library-mode quickstart and the import of run_pipeline in pipeline_runners.py. Trace the import chain to decorators.py, where Manager() is created, and reproduce the failure on macOS using the provided minimal example. Done means the supported macOS behavior is resolved or the quickstart clearly documents the platform limitation and expected setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.