[Coverage] TypeError: _ModuleStackTracer.__init__() missing 1 required positional argument: 'scope_root'

HuggingFace/diffusers-runwayml/stable-diffusion-v1-5/text_encoder.0

2024-08-11 18:22:57.661 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py:218 - Command: /usr/bin/python torch/reproduce_correctness-torchtensorrtcompilerunner.py --batch_dim '0' --results_path '/tmp/tmp7sfhcf5s' --runner_name 'TorchTensorRTCompile' --input_metadata '{"metadata": [{"name": "input__0", "shape": (-1, 4, 64, 64), "dtype": "float32"}, {"name": "input__1", "shape": (), "dtype": "int64"}, {"name": "input__2", "shape": (-1, 77, 768), "dtype": "float32"}], "pytree_metadata": {"metadata": ("input__0", "input__1", {"added_cond_kwargs": None, "cross_attention_kwargs": None, "encoder_hidden_states": "input__2", "return_dict": False, "timestep_cond": None}), "tensor_type": "torch"}, "is_legacy": False}' --output_metadata '{"metadata": [{"name": "output__0", "shape": (-1, 4, 64, 64), "dtype": "float16"}], "pytree_metadata": {"metadata": ("output__0",), "tensor_type": "torch"}, "is_legacy": False}' --runner_config '{"autocast": True, "inference_mode": True}'
2024-08-11 18:22:58.388 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/runners/torch.py:378 - Using torch.compile with config: fullgraph=False, dynamic=False, backend=torch_tensorrt, mode=None, options={'truncate_long_and_double': True, 'enabled_precisions': {torch.float32, torch.float16}}
2024-08-11 18:23:05.900 | WARNING  | MainProcess | /usr/local/lib/python3.10/dist-packages/torch_tensorrt/dynamo/_compiler.py:297 - Node _param_constant6 of op type get_attr does not have metadata. This could sometimes lead to undefined behavior.
2024-08-11 18:23:05.901 | WARNING  | MainProcess | /usr/local/lib/python3.10/dist-packages/torch_tensorrt/dynamo/_compiler.py:306 - Some nodes do not have metadata (shape and dtype information). This could lead to problems sometimes if the graph has PyTorch and TensorRT segments.
WARNING: [Torch-TensorRT] - Using default stream in enqueueV3() may lead to performance issues due to additional calls to cudaStreamSynchronize() by TensorRT to ensure correct synchronization. Please use non-default stream instead.
2024-08-11 18:29:35.985 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:149 - Execution time: 398.33[s]
2024-08-11 18:29:36.020 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:116 - ============== Command 'Correctness' started ===============
2024-08-11 18:29:36.022 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py:117 - Correctness test for: Format.TORCHSCRIPT <class 'model_navigator.runners.torch.TorchScriptCUDARunner'> started.
2024-08-11 18:29:36.022 | WARNING  | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py:122 - Model: '/root/.cache/model_navigator/unet/0/torchscript-script/model.pt' not found, command skipped.
2024-08-11 18:29:36.023 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:149 - Execution time: 0.00[s]
2024-08-11 18:29:36.051 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:116 - ============== Command 'Correctness' started ===============
2024-08-11 18:29:36.053 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py:117 - Correctness test for: Format.TORCHSCRIPT <class 'model_navigator.runners.torch.TorchScriptCUDARunner'> started.
2024-08-11 18:29:36.053 | WARNING  | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py:122 - Model: '/root/.cache/model_navigator/unet/0/torchscript-trace/model.pt' not found, command skipped.
2024-08-11 18:29:36.054 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:149 - Execution time: 0.00[s]
2024-08-11 18:29:36.082 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:116 - ============== Command 'Correctness' started ===============
2024-08-11 18:29:36.084 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py:117 - Correctness test for: Format.TORCH_EXPORTEDPROGRAM <class 'model_navigator.runners.torch.TorchExportedProgramCUDARunner'> started.
2024-08-11 18:29:36.085 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py:218 - Command: /usr/bin/python torch-exportedprogram/reproduce_correctness-torchexportedprogramcudarunner.py --batch_dim '0' --results_path '/tmp/tmpekqsm7hx' --runner_name 'TorchExportedProgramCUDA' --input_metadata '{"metadata": [{"name": "input__0", "shape": (-1, 4, 64, 64), "dtype": "float32"}, {"name": "input__1", "shape": (), "dtype": "int64"}, {"name": "input__2", "shape": (-1, 77, 768), "dtype": "float32"}], "pytree_metadata": {"metadata": ("input__0", "input__1", {"added_cond_kwargs": None, "cross_attention_kwargs": None, "encoder_hidden_states": "input__2", "return_dict": False, "timestep_cond": None}), "tensor_type": "torch"}, "is_legacy": False}' --output_metadata '{"metadata": [{"name": "output__0", "shape": (-1, 4, 64, 64), "dtype": "float16"}], "pytree_metadata": {"metadata": ("output__0",), "tensor_type": "torch"}, "is_legacy": False}' --runner_config '{"autocast": True, "inference_mode": True}' --model_path 'torch-exportedprogram/model.pt2'
/usr/local/lib/python3.10/dist-packages/modelopt/torch/quantization/tensor_quant.py:84: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  scaled_e4m3_abstract = torch.library.impl_abstract("trt::quantize_fp8")(
[WARNING  | py.warnings        ]: /usr/local/lib/python3.10/dist-packages/tritonclient/grpc/service_pb2_grpc.py:21: RuntimeWarning: The grpc package installed is at version 1.62.1, but the generated code in grpc_service_pb2_grpc.py depends on grpcio>=1.64.1. Please upgrade your grpc module to grpcio>=1.64.1 or downgrade your generated code using grpcio-tools<=1.62.1. This warning will become an error in 1.65.0, scheduled for release on June 25, 2024.
  warnings.warn(
2024-08-11 18:29:41.026 | ERROR    | Process-4 | /usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py:164 - Command exited with error: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 156, in _execute_function
    fire.Fire(func, unwrapped_args)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness_script.py", line 91, in correctness
    with runner:
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/runners/base.py", line 222, in __enter__
    self.activate()
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/runners/base.py", line 258, in activate
    self.activate_impl()
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/runners/torch.py", line 438, in activate_impl
    self._loaded_model = torch.load(str(self._model), map_location=self.device)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1036, in load
    return _load(opened_zipfile,
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1460, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 127, in reduce_graph_module
    return _deserialize_graph_module(forward, body)
  File "/usr/local/lib/python3.10/dist-packages/torch/fx/graph_module.py", line 191, in _deserialize_graph_module
    graph = KeepModules().trace(com, **tracer_extras)
TypeError: _ModuleStackTracer.__init__() missing 1 required positional argument: 'scope_root'
. Command to reproduce error: /bin/bash torch-exportedprogram/reproduce_correctness-torchexportedprogramcudarunner.sh
2024-08-11 18:29:41.925 | WARNING  | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:132 - Command finished with ModelNavigatorUserInputError. The error is considered as external error. Usually caused by incompatibilities between the model and the target formats and/or runtimes. Please review the command output.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py", line 122, in _execute_unit
    command_output = execution_unit.command().run(
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/base.py", line 126, in run
    output = self._run(*args, **_filter_dict_for_func(kwargs, self._run))
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/correctness/correctness.py", line 150, in _run
    context.execute_python_script(
  File "/usr/local/lib/python3.10/dist-packages/model_navigator/commands/execution_context.py", line 138, in execute_python_script
    raise ModelNavigatorUserInputError(
model_navigator.exceptions.ModelNavigatorUserInputError: Process exited with 1. Check previous logs for errors.
2024-08-11 18:29:41.925 | INFO     | MainProcess | /usr/local/lib/python3.10/dist-packages/model_navigator/pipelines/pipeline.py:149 - Execution time: 5.84[s]

Start with torch-exportedprogram/reproduce_correctness-torchexportedprogramcudarunner.sh and the corresponding reproduce_correctness-torchexportedprogramcudarunner.py command shown in the report. Review the model loading path in model_navigator/runners/torch.py and the PyTorch FX deserialization traceback; done means the correctness command no longer exits with the reported TypeError.