[Arm backend] can't convert yolo models with aot_arm_compiler
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🐛 Describe the bug
I'm using the latest release version v0.6.And I want to convert yolo models with arm backend.
I modified the examples as bellow, and convert the model with the command:
python3 -m examples.arm.aot_arm_compiler --model_name="yolo" --delegate --quantize --so_library="$(find arm_test -name libquantized_ops_aot_lib.so)"
diff --git a/examples/arm/aot_arm_compiler.py b/examples/arm/aot_arm_compiler.py
index 5fb12342a..a9f216cc2 100644
--- a/examples/arm/aot_arm_compiler.py
+++ b/examples/arm/aot_arm_compiler.py
@@ -29,6 +29,7 @@ from executorch.backends.arm.quantizer.arm_quantizer import (
get_symmetric_quantization_config,
TOSAQuantizer,
)
+from ultralytics import YOLO
from executorch.backends.arm.tosa_partitioner import TOSAPartitioner
from executorch.backends.arm.tosa_specification import TosaSpecification
@@ -82,6 +83,10 @@ def get_model_and_inputs_from_name(
model = models[model_name]()
if example_inputs is None:
example_inputs = models[model_name].example_input
+ elif model_name == "yolo":
+ model = YOLO("yolov5s.pt").model
+ example_inputs = (torch.randn(1, 3, 640, 640),)
elif model_name in MODEL_NAME_TO_MODEL.keys():
logging.warning(
@@ -648,7 +653,7 @@ def to_edge_TOSA_delegate(
)
model_int8 = model
# Wrap quantized model back into an exported_program
- exported_program = torch.export.export_for_training(model, example_inputs)
+ exported_program = torch.export.export_for_training(model, example_inputs, strict=False)
INFO:executorch.backends.arm.tosa_partitioner:TOSAPartitioner::partition
INFO:executorch.backends.arm.tosa_partitioner:Partitioning for EthosUBackend: TOSA-0.80+BI+u55
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/bamboo/latest/executorch/examples/arm/aot_arm_compiler.py", line 696, in <module>
model_int8, edge = to_edge_TOSA_delegate(exported_program, args, model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/examples/arm/aot_arm_compiler.py", line 668, in to_edge_TOSA_delegate
edge = to_edge_transform_and_lower(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 107, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 1209, in to_edge_transform_and_lower
edge_manager = edge_manager.to_backend({name: curr_partitioner})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 107, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/program/_program.py", line 1471, in to_backend
new_edge_programs[name] = to_backend(program, partitioner[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/functools.py", line 909, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/backend/backend_api.py", line 377, in _
partitioner_result = partitioner_instance(fake_edge_program)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/exir/backend/partitioner.py", line 66, in __call__
return self.partition(exported_program)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/tosa_partitioner.py", line 90, in partition
partition_list = capability_partitioner.propose_partitions()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/infra/partitioner.py", line 221, in propose_partitions
if self.__is_node_supported(node) and node not in assignment:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/infra/partitioner.py", line 84, in __is_node_supported
return self.operator_support.is_node_supported(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 153, in is_node_supported
return is_node_supported(submodules, node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 166, in _chain
return all(x.is_node_supported(submods, node) for x in op_support)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/torch/fx/passes/operator_support.py", line 166, in <genexpr>
return all(x.is_node_supported(submods, node) for x in op_support)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/operator_support/tosa_supported_operators.py", line 471, in is_node_supported
tensor = get_first_fake_tensor(input_node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bamboo/latest/executorch/.venv/lib/python3.12/site-packages/executorch/backends/arm/_passes/arm_pass_utils.py", line 158, in get_first_fake_tensor
assert isinstance(
^^^^^^^^^^^
AssertionError: Found zuf38 in meta["val"] of _local_scalar_dense_2, expected to find FakeTensor.
Versions
commit e67ef3b149688b4085605495690d0d5c6420b45a (HEAD -> main, tag: v0.6.0-rc9, tag: v0.6.0)
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command from examples/arm/aot_arm_compiler.py with the YOLO model and inspect the TOSA partitioning path. Start at tosa_supported_operators.py and _passes/arm_pass_utils.py, focusing on the _local_scalar_dense_2 metadata assertion. Done means the reported YOLO conversion no longer fails at this assertion and the ARM compilation flow completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- embedded-iot, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100