pytorch / pytorch/executorch

executorch model Inference time is higher than the torch model

Open
#10,297 38 comments 1 reaction 1 assignee View on GitHub

@GregoryComer is already working on this.

Since Apr 18, 2025.

module: xnnpack
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

I have a model object, converted it to pte model with xnn backend using below:
exported_graph = export(model, inp) # Core Aten graph torch.export.save(exported_program, 'model.pt2') edge = to_edge(exported_graph) # Edge Dialect edge_delegated = edge.to_backend(XnnpackPartitioner()) #using xnnbackend executorch_program = edge_delegated.to_executorch() # with open("model.pte", "wb") as file: file.write(executorch_program.buffer)

then used it in C++ frontend to run my llm application in similar lines of example
Application has executorch in the third-party folder.

I want help with two things,

  1. executorch runtime is taking more time ~16 seconds, where as torch inference would run in around 1.3 seconds.
    I want some help in improving the inference times. I can share pte graph log in private if needed If fusing ops / removing ops will help in reducing time.
    <bound method EdgeProgramManager.exported_program of <executorch.exir.program._program.EdgeProgramManager object at 0x77ae4cff0640>>
    graph():
  2. I want to know If I can selectively build based on the ops needed by graph. I could see that my exported graph (pt2) has some around 16 aten ops.
    How should I delegate it to backend as it may have different operator set?
    or is it taken care by selective print gen_selected_ops function based on arguments given?

How ever I'm unable to build selectively only based on the ops, I would appreciate some help here too. Below is part of my cmakelist to add selected ops and include library to target
set(_kernel_lib) gen_selected_ops(LIB_NAME "select_build_lib" "" ROOT_OPS "aten::add.out" INCLUDE_ALL_OPS "OFF") generate_bindings_for_kernels(LIB_NAME "select_build_lib" FUNCTIONS_YAML ${EXECUTORCH_ROOT}/kernels/portable/functions.yaml) gen_operators_lib(LIB_NAME "select_build_lib" KERNEL_LIBS ${_kernel_lib} DEPS executorch) target_link_libraries( my_app PRIVATE executorch extension_module_static extension_tensor xnnpack_backend select_build_lib)

cc @digantdesai @mcr229 @cbilgin

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.