[Coverage] NAN output for BasicUNet and Qwen-7B-Chat
Open
Nobody has claimed this yet.
story: LLM & Generative AI
- Dominant language
- Python
- Stars
- 3k
- Forks
- 410
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 78
Description
2024-08-30 23:51:44.645 | ERROR | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness_script.py:106 - Comparison output output__0 contains NaN
2024-08-30 23:51:45.205 | INFO | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:128 - 1
monai.networks.nets.basic_unet.BasicUNet: Validating model torch on
TorchTensorRTCompile backend FAIL
2024-08-30 23:51:45.213 | WARNING | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:131 - Command finished with ModelNavigatorUserInputError. The error is considered as external error. Usually caused by incompatibilities between the model and the target formats and/or runtimes. Please review the command output.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py", line 150, in _run
context.execute_python_script(
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 142, in execute_python_script
self._execute_function(func, unwrapped_args, allow_failure, cmd)
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 156, in _execute_function
fire.Fire(func, unwrapped_args)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness_script.py", line 107, in correctness
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py", line 121, in _execute_unit
command_output = execution_unit.command().run(
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/base.py", line 127, in run
output = self._run(*args, **_filter_dict_for_func(kwargs, self._run))
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py", line 125, in _run
with ExecutionContext(
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 102, in __exit__
raise ModelNavigatorUserInputError(message=message)
model_navigator.exceptions.ModelNavigatorUserInputError: 1
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py", line 150, in _run
context.execute_python_script(
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 142, in execute_python_script
self._execute_function(func, unwrapped_args, allow_failure, cmd)
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 156, in _execute_function
fire.Fire(func, unwrapped_args)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness_script.py", line 107, in correctness
sys.exit(1)
2024-08-30 23:51:45.214 | INFO | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:148 - Execution time: 280.05[s]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the coverage run for BasicUNet and Qwen-7B-Chat from the reported NaN output. Inspect model_navigator/commands/correctness/correctness_script.py and model_navigator/pipelines/pipeline.py, then identify whether the NaN originates in correctness comparison or TorchTensorRT compilation; done means the affected models complete validation without NaN output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100