mapillary / mapillary/OpenSfM

Deadlock in pytest tests

Open
#984 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.8k
Forks
899
PR merge metrics
No merged PRs in 30d

Description

I encountered an issue when building this software with the Nix package manager on NixOS. Our package uses commit 536b6e1414c8a93f0815dbae85d03749daaa5432 because there is no recent release.

nix-build '<nixpkgs>' -A python310Packages.opensfm --check

The build is often stuck at a very specific point:

...
opensfm/test/test_dense.py ..                                            [ 55%]
opensfm/test/test_geo.py .....                                           [ 57%]
opensfm/test/test_geometry.py ....                                       [ 59%]
opensfm/test/test_io.py .........                                        [ 62%]
opensfm/test/test_matching.py ......                                     [ 65%]
opensfm/test/test_multiview.py .........                                 [ 68%]
opensfm/test/test_pairs_selection.py .........                           [ 72%]
opensfm/test/test_reconstruction_alignment.py ........                   [ 75%]
opensfm/test/test_reconstruction_incremental.py Terminated

So i have to kill the build process. I waited over 3.5 hours once!

Maybe the last test creates a deadlock?

When i look at the process with strace, it only does sched_yield calls.

I have seen similar issues with other python packages doing pytest tests and only doing futex calls at some point.

This seem to happen especially when multiple python packages are built at the same time.

Downstream issue: https://github.com/NixOS/nixpkgs/issues/217247
Our package definition: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/python-modules/opensfm/default.nix

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Nix check with nix-build '<nixpkgs>' -A python310Packages.opensfm --check, then focus on opensfm/test/test_reconstruction_incremental.py, where the run stops. Compare behavior when Python packages build concurrently and inspect the process with strace. Done means the pytest suite completes reliably instead of hanging at that test.

Written by the indexing model from the issue text.

Assessment

Tech stack
nixos, python
Domain
build-system, computer-vision, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.