Project-MONAI / Project-MONAI/MONAILabel
Run monai-label in kubernetes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 891
- Forks
- 269
- Avg merge
- 15h 41m
- Merged PRs (30d)
- 1
Description
Hello,
is it possible to install monai-label on kubernetes?
I'm trying to make it work via a classic deployment but I'm encountering the following problems in my pods:
[k8sgpu-01|monai-dev] imarson_adm@pc12:~/monai$ k logs monai-label-6cd85976c5-6rjtl
=============
== PyTorch ==
NVIDIA Release 24.03 (build 85286408)
PyTorch Version 2.3.0a0+40ec155e58
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
NOTE: CUDA Forward Compatibility mode ENABLED.
Using CUDA 12.4 driver version 550.54.14 with kernel driver version 535.129.03.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.
NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for PyTorch. NVIDIA recommends the use of the following flags:
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...
=> And my pod is in crashLoopBackOff
Do You have some advice for using it?
Thank you
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Kubernetes deployment configuration and the pod logs shown in the issue, focusing on why the MONAI Label pod enters CrashLoopBackOff. Reproduce the deployment if possible and determine the missing configuration or runtime failure. Done means MONAI Label starts successfully in the Kubernetes pod.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, pytorch
- Domain
- infrastructure, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100