FedML-AI / FedML-AI/FedML

The compatibility issues of Nvidia Jetson

Open
#1,854 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.1k
Forks
765
PR merge metrics
No merged PRs in 30d

Description

# I think that the installation instructions for the FedML framework on the official website maybe need to be updated

- Initially, I followed the tutorial on the [official website](https://doc.fedml.ai/open-source/installation/jetson) to download the FedML framework.
**`docker pull fedml/fedml:latest-nvidia-jetson-l4t-ml-r35.1.0-py3`**
The version of FedML downloaded from this mirror source was 0.7.6, but I don't remember the specific version.This version can run some official examples well, such as:([fedcv](https://github.com/FedML-AI/FedML/tree/3d93c76018578f0399fa9716fa98a69061de951c/python/examples/federate/prebuilt_jobs/fedcv))

- However, an error occurred when I ran my own federated learning training for anomaly detection in a computer vision surveillance scenario.I sought help from the community, and someone advised me to update my version of FedML as mine was too old. So, I used the command `pip install -U fedml` to update the version of FedML.

- After updating the version, when I ran my code again, I encountered a new error related to the architecture of numpy.[This is the log of my project.](https://nexus.fedml.ai/federate/octopus/collaborator/runDetail?projectId=3559&groupId=6502&runId=27082)

> ![fedml1](https://github.com/FedML-AI/FedML/assets/115770276/dbc27dcb-2cfd-4b8a-905b-a01da9f6b4ad)

> This image shows the numpy issue that occurred when my Jetson device was connected to MLOps.

# The error I discovered

### After updating to FedML version 0.8.17, I checked the supported versions of the related libraries and found that both the torch library and the torchvision library versions are incompatible.

- The version of the mirror source pulled from the FedML official website is r35.1.0-py3.
![image](https://github.com/FedML-AI/FedML/assets/115770276/101fc4bc-fd0d-4683-a73f-13bd55eea73a)

- Here are the corresponding [versions of the libraries for the r35.1.0-py3 mirror source:](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-ml)
![image](https://github.com/FedML-AI/FedML/assets/115770276/a02f0750-8d7e-4d22-afa7-7bd307b33f4d)

- For `fedml==0.8.17`, the corresponding [versions ](https://github.com/FedML-AI/FedML/blob/3d93c76018578f0399fa9716fa98a69061de951c/python/setup.py#L32)of torch and torchvision are:
```
"torch>=1.13.1",
"torchvision>=0.14.1",
```
#### I believe that the numpy error is likely caused by the versions of torch and torchvision within the mirror source.

# My Solution

### First, I re-flashed my NVIDIA JETSON device with the latest JetPack version, JetPack 5.1.2.Then, I attempted to use `docker pull fedml/fedml:latest-nvidia-jetson-l4t-ml-r36.2.0-py3`to pull the latest mirror source and download FedML.But I was unable to download using this command line.

- I first pulled the latest version of the mirror source using the command
`sudo docker pull nvcr.io/nvidia/l4t-ml:r36.2.0-py3`
- Then, I opened an interactive container by running the command
`sudo docker run -it nvcr.io/nvidia/l4t-ml:r36.2.0-py3 /bin/bash`
- downloaded the latest version of the FedML framework by using the command
`pip install fedml==0.8.17`
### updated the contents of the container to create a new image:
- Open a new terminal and enter the following command:
`sudo docker ps`
- Copy the ID of the container and enter the following command:
`sudo docker commit `

### To enter the newly saved image, for example, if the name of the new image is "fedml"
- Please enter the following command:
`sudo docker run -t -i --runtime nvidia fedml /bin/bash`
- Then connect to MLOps using the command:
`fedml login $account_id`
- If You encounter an issue where "/bin/sh: python: not found" error appears:
```
ls -l /usr/bin/python*
ln -s /usr/bin/python3 /usr/bin/python
```
**This is because the path /usr/bin/python has been overwritten by Python 2 and Python 3.**

# Subsequent issues encountered
### After updating the environment, I did not encounter any further issues with numpy. But I encountered a new problem.
![fedml2](https://github.com/FedML-AI/FedML/assets/115770276/6323f01d-db9b-4fff-861a-bb835efc33d5)
- Please provide [the error log file](https://nexus.fedml.ai/federate/octopus/collaborator/runDetail?projectId=3559&groupId=6502&runId=27103) for further assistance.
- This doesn't seem to be an issue with my own code because I encountered the same error when running [the official example.](https://github.com/FedML-AI/FedML/tree/3d93c76018578f0399fa9716fa98a69061de951c/python/examples/federate/cross_silo/mqtt_s3_fedavg_mnist_lr_example)
- Here are the error logs for the two issues:
[My own example.](https://nexus.fedml.ai/federate/octopus/collaborator/runDetail?projectId=3559&groupId=6502&runId=27103)
[the official example](https://nexus.fedml.ai/federate/octopus/collaborator/runDetail?projectId=3489&groupId=6502&runId=27105)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.