conda-forge / conda-forge/opencv-feedstock
Importing opencv when running pytorch parallel workloads leads to hangs when using opencv from conda
- Dominant language
- Shell
- Stars
- 70
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
### Solution to issue cannot be found in the documentation.
- [x] I checked the documentation.
### Issue
opencv and PyTorch seem to interact poorly when using the cpu backend of PyTorch. The following program exposes the issue. It is a simple convolutional network that trains on random data, run 8 times in parallel. Its just there to simulate work, it doesn’t do anything meaningful.:
```
import multiprocessing as mp
# import cv2
# Function to train the model (wrapped for multiprocessing)
def train_model(rank):
print(f"Starting process {rank}...")
import torch
import torch.nn as nn
import torch.optim as optim
# Define a small CNN model
class SmallCNN(nn.Module):
def __init__(self):
super(SmallCNN, self).__init__()
self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1)
self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
self.fc = nn.Linear(64 * 8 * 8, 10) # Fully connected layer for classification
def forward(self, x):
x = torch.relu(self.conv1(x))
x = torch.relu(self.conv2(x))
x = torch.relu(self.conv3(x))
x = torch.flatten(x, start_dim=1) # Flatten before FC layer
x = self.fc(x)
return x
batch_size = 64
num_batches = 100 # Number of random batches per epoch
num_epochs = 10
lr = 0.01 # Learning rate
# Create model, loss function, and optimizer (No CUDA)
model = torch.compile(SmallCNN())
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
for epoch in range(num_epochs):
total_loss = 0
for _ in range(num_batches):
inputs = torch.randn(batch_size, 3, 8, 8)
labels = torch.randint(0, 10, (batch_size,)) # Random labels (10 classes)
# Forward pass
outputs = model(inputs)
loss = criterion(outputs, labels)
# Backward pass
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f"Process {rank} - Epoch {epoch+1}/{num_epochs}, Loss: {total_loss / num_batches:.4f}")
print(f"Process {rank} complete! 🎉")
if __name__ == "__main__":
mp.set_start_method("spawn") # Ensures compatibility
num_processes = 10 # Launch 10 processes
processes = []
for rank in range(num_processes):
p = mp.Process(target=train_model, args=(rank,))
p.start()
processes.append(p)
for p in processes:
p.join()
print("All processes finished successfully! ✅")
```
Uncomment line 2 to reproduce the issue.
Enable openmp before executing:
`export OMP_NUM_THREADS=10 OPENBLAS_NUM_THREADS=10 MKL_NUM_THREADS=10 VECLIB_MAXIMUM_THREADS=10 NUMEXPR_NUM_THREADS=10 DASK_NUM_THREADS=10`
If opencv is not imported: the program runs, it takes a little while but it runs through the epochs. Depending on the machine this is running, you may have to adapt number of threads or processes to not completely lock it up.
If opencv is imported, the program locks up.
It does appear that importing opencv changes some openmp related state such that using openmp with torch completely breaks torch. In our workloads, I noticed very evident traces of memory corruption (tensors changing values even though they really should not). Torch hanging is the easiest to reproduce, so that’s what this sample reproduces.
If I install opencv from pip (but keep torch on Conda), it works. So it likely has something to do the way opencv is built in Conda.
Not sure the issue is with pytorch or with opencv. May be cross posting may make sense to make the pytorch people aware of the issue?
### Installed packages
```shell
To reproduce, create the env as follows:
conda create -n memory_corruption_debug python=3.10
conda activate memory_corruption_debug
mamba install pytorch=2.4*=*cpu* protobuf=4 opencv=4.10*=*headless*
I am not sure protobuf=4 is needed, it just replicates the environment we are using
```
### Environment info
```shell
active environment : shrek
active env location : /opt/miniconda/envs/memory_corruption_debug
shell level : 1
user config file : /root/.condarc
populated config files : /opt/miniconda/.condarc
conda version : 24.11.3
conda-build version : not installed
python version : 3.11.11.final.0
solver : libmamba (default)
virtual packages : __archspec=1=zen2
__conda=24.11.3=0
__cuda=12.6=0
__glibc=2.35=0
__linux=5.10.192=0
__unix=0=0
base environment : /opt/miniconda (writable)
conda av data dir : /opt/miniconda/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
package cache : /opt/miniconda/pkgs
/root/.conda/pkgs
envs directories : /opt/miniconda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/24.11.3 requests/2.31.0 CPython/3.11.11 Linux/5.10.192-183.736.amzn2.x86_64 ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5
UID:GID : 0:0
netrc file : None
offline mode : False
```
Contributor guide
Assessment
This issue has not been assessed yet.