apple / apple/coremltools

Question re. ANE Usage with Flexible Input Shapes

Open
#1,764 4 comments 2 reactions 0 assignees View on GitHub
Core ML Framework question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question

Not sure if this is a framework issue, or one with `coremltools`. My hunch is the latter, so I'm asking here.

I've exported a model that requires a flexible input shape, and set the default shape to `1`. This model doesn't use the ANE at all, and only runs on CPU.

Out of curiosity, I set the input shape to be fixed to `1` to see if the model would run faster. This model uses the GPU / ANE and is _significantly faster_. Does this mean that ANE usage is out of the window with flexible input shapes, or is there scope to redefine the model to allow it to use the ANE with flexible shapes too?

Unfortunately, I cannot share the model definition publicly.

### Fixed input shape:

![CleanShot 2023-02-09 at 18 46 12](https://user-images.githubusercontent.com/17111474/217884623-426d746e-2587-4272-9a1f-2a5199059dd8.png)

### Flexible input shape:

![CleanShot 2023-02-09 at 18 46 15](https://user-images.githubusercontent.com/17111474/217884785-e56702c2-2a5f-48bb-9818-c080bce8c7fc.png)

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by reproducing the reported performance difference with equivalent fixed and flexible input shapes, then investigate whether coremltools or the exported model controls ANE eligibility; done means determining whether flexible shapes can use the ANE and documenting the supported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.