pytorch / pytorch/executorch

[Arm backend] can't convert yolo models with aot_arm_compiler

Open
#12,237 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module: arm triaged
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

🐛 Describe the bug

I'm using the latest release version v0.6.And I want to convert yolo models with arm backend.
I modified the examples as bellow, and convert the model with the command:
python3 -m examples.arm.aot_arm_compiler --model_name="yolo" --delegate --quantize --so_library="$(find arm_test -name libquantized_ops_aot_lib.so)"

    diff --git a/examples/arm/aot_arm_compiler.py b/examples/arm/aot_arm_compiler.py
    index 5fb12342a..a9f216cc2 100644
    --- a/examples/arm/aot_arm_compiler.py
    +++ b/examples/arm/aot_arm_compiler.py
    @@ -29,6 +29,7 @@ from executorch.backends.arm.quantizer.arm_quantizer import (
         get_symmetric_quantization_config,
         TOSAQuantizer,
     )
    +from ultralytics import YOLO
     from executorch.backends.arm.tosa_partitioner import TOSAPartitioner
     from executorch.backends.arm.tosa_specification import TosaSpecification
    
    @@ -82,6 +83,10 @@ def get_model_and_inputs_from_name(
             model = models[model_name]()
             if example_inputs is None:
                 example_inputs = models[model_name].example_input
    +    elif model_name == "yolo":
    +        model = YOLO("yolov5s.pt").model
    +        example_inputs = (torch.randn(1, 3, 640, 640),)
     
         elif model_name in MODEL_NAME_TO_MODEL.keys():
             logging.warning(
    @@ -648,7 +653,7 @@ def to_edge_TOSA_delegate(
             )
             model_int8 = model
             # Wrap quantized model back into an exported_program
    -        exported_program = torch.export.export_for_training(model, example_inputs)
    +        exported_program = torch.export.export_for_training(model, example_inputs, strict=False)
    

INFO:executorch.backends.arm.tosa_partitioner:TOSAPartitioner::partition
INFO:executorch.backends.arm.tosa_partitioner:Partitioning for EthosUBackend: TOSA-0.80+BI+u55
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/bamboo/latest/executorch/examples/arm/aot_arm_compiler.py", line 696, in <module>
    model_int8, edge = to_edge_TOSA_delegate(exported_program, args, model)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/examples/arm/aot_arm_compiler.py", line 668, in to_edge_TOSA_delegate
    edge = to_edge_transform_and_lower(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 107, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 1209, in to_edge_transform_and_lower
    edge_manager = edge_manager.to_backend({name: curr_partitioner})
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 107, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 1471, in to_backend
    new_edge_programs[name] = to_backend(program, partitioner[name])
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 909, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/backend/backend_api.py", line 377, in _
    partitioner_result = partitioner_instance(fake_edge_program)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/backend/partitioner.py", line 66, in __call__
    return self.partition(exported_program)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/tosa_partitioner.py", line 90, in partition
    partition_list = capability_partitioner.propose_partitions()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/infra/partitioner.py", line 221, in propose_partitions
    if self.__is_node_supported(node) and node not in assignment:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/infra/partitioner.py", line 84, in __is_node_supported
    return self.operator_support.is_node_supported(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 153, in is_node_supported
    return is_node_supported(submodules, node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 166, in _chain
    return all(x.is_node_supported(submods, node) for x in op_support)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 166, in <genexpr>
    return all(x.is_node_supported(submods, node) for x in op_support)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/operator_support/tosa_supported_operators.py", line 471, in is_node_supported
    tensor = get_first_fake_tensor(input_node)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/_passes/arm_pass_utils.py", line 158, in get_first_fake_tensor
    assert isinstance(
           ^^^^^^^^^^^
AssertionError: Found zuf38 in meta["val"] of _local_scalar_dense_2, expected to find FakeTensor.

Versions

commit e67ef3b149688b4085605495690d0d5c6420b45a (HEAD -> main, tag: v0.6.0-rc9, tag: v0.6.0)

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command from examples/arm/aot_arm_compiler.py with the YOLO model and inspect the TOSA partitioning path. Start at tosa_supported_operators.py and _passes/arm_pass_utils.py, focusing on the _local_scalar_dense_2 metadata assertion. Done means the reported YOLO conversion no longer fails at this assertion and the ARM compilation flow completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
embedded-iot, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.