executorch model Inference time is higher than the torch model
@GregoryComer is already working on this.
Since Apr 18, 2025.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
I have a model object, converted it to pte model with xnn backend using below:
exported_graph = export(model, inp) # Core Aten graph torch.export.save(exported_program, 'model.pt2') edge = to_edge(exported_graph) # Edge Dialect edge_delegated = edge.to_backend(XnnpackPartitioner()) #using xnnbackend executorch_program = edge_delegated.to_executorch() # with open("model.pte", "wb") as file: file.write(executorch_program.buffer)
then used it in C++ frontend to run my llm application in similar lines of example
Application has executorch in the third-party folder.
I want help with two things,
- executorch runtime is taking more time ~16 seconds, where as torch inference would run in around 1.3 seconds.
I want some help in improving the inference times. I can share pte graph log in private if needed If fusing ops / removing ops will help in reducing time.
<bound method EdgeProgramManager.exported_program of <executorch.exir.program._program.EdgeProgramManager object at 0x77ae4cff0640>>
graph(): - I want to know If I can selectively build based on the ops needed by graph. I could see that my exported graph (pt2) has some around 16 aten ops.
How should I delegate it to backend as it may have different operator set?
or is it taken care by selective print gen_selected_ops function based on arguments given?
How ever I'm unable to build selectively only based on the ops, I would appreciate some help here too. Below is part of my cmakelist to add selected ops and include library to target
set(_kernel_lib) gen_selected_ops(LIB_NAME "select_build_lib" "" ROOT_OPS "aten::add.out" INCLUDE_ALL_OPS "OFF") generate_bindings_for_kernels(LIB_NAME "select_build_lib" FUNCTIONS_YAML ${EXECUTORCH_ROOT}/kernels/portable/functions.yaml) gen_operators_lib(LIB_NAME "select_build_lib" KERNEL_LIBS ${_kernel_lib} DEPS executorch) target_link_libraries( my_app PRIVATE executorch extension_module_static extension_tensor xnnpack_backend select_build_lib)
cc @digantdesai @mcr229 @cbilgin
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.