model.eval causing nan values
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 298
- Forks
- 26
- Avg merge
- 22h 44m
- Merged PRs (30d)
- 1
Description
Thanks for sharing your work !
@Yukichiii @yuxiaoguo I tried to test your code on Semantic3D :
In validation step, i get "nan" value in output.
- I checked points cloud data, and there is no "nan" in npy files
- I used this : print('Nan value .... ???? ', [k for k, v in model.named_parameters() if any(torch.isnan(v.ravel()))]). There is no nan values in train (model.train()) and validation (model.eval())
- torch.where( torch.isnan(coord) == True), torch.where( torch.isnan(feat) == True), torch.where( torch.isnan(batch) == True) return empty list
- So : neither nan value in data nor weigths/biais -> however output filled with nan
Do you have any idea where the problem could come from (layer norm, ....) ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the validation run on Semantic3D and trace the validation path around model.eval(), checking intermediate outputs rather than only parameters and input tensors. The issue mentions layer normalization as a possible area, but no file or test is identified. Done means locating the source of the NaN output and confirming that validation produces finite outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100