MrGiovanni / MrGiovanni/ModelsGenesis

something about cpu explosion

Open
#48 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
791
Forks
141
PR merge metrics
No merged PRs in 30d

Description

hi, Giovanni, I just re-implemented the MG-2D on my dataset and met with some other bug for cpu explosion.

When I used MG for pre-training, I found that the utilization rate of the CPU would increase strangely, but the utilization rate of the GPU fluctuated greatly, and the buzzer of the machine PC would alarm soon. After several rounds of debugging, I found that it was this piece of code that caused the CPU utilization to sharply increase to 4000%+!!!!
https://github.com/MrGiovanni/ModelsGenesis/blob/13932523d1754a3180301448f4bfd5274a1c9f41/pytorch/utils.py#L45
I don't know why, but after changing np.dot in function : nonlinear_transformation and related calculations to torch, the CPU utilization has returned to normal (200%+).

Maybe this is a special case in my machine, and it has no reference value.
Thanks. :-)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at pytorch/utils.py line 45, in nonlinear_transformation, and inspect the np.dot call and related calculations. Reproduce the MG pre-training case if possible, then compare the CPU and GPU utilization with the reported torch-based calculations. Done means the transformation no longer causes extreme CPU utilization while preserving the expected pre-training behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.