Strange issue: Converting RealESRGAN to CoreML model, ML Program format performs significantly slower than Neural Network format
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
Hi everyone,
I'm currently learning about machine learning in the Apple ecosystem. I tried converting the [RealESRGAN](https://github.com/xinntao/Real-ESRGAN) super-resolution model to a CoreML model and found that the ML Program format performs much slower than the Neural Network format, especially with load times being over 10 times longer.
In theory, the newer ML Program format should perform better, but that's not the case here. I'm not sure where the issue lies. Here's the [script](https://gist.github.com/itcook/e39f921eb50913988f4f762f33a80d8c) I used for the conversion—I'd really appreciate any help if someone can provide insights.
Additionally, after setting the input tensor's width and height to fixed dimensions, the model's tracing time became extremely long, whereas previously, using RangDim to set flexible shapes didn't exhibit this issue (the sample inputs were almost identical). This also puzzles me...
Furthermore, does a model with fixed-shape inputs generally perform better than one with flexible shapes? My understanding is that, in terms of performance: fixed-shape input > enumerated fixed-shapes input > flexible-shape input. Is this correct?
Contributor guide
Research direction
Begin with the linked conversion script and its fixed- versus flexible-shape settings; compare the generated ML Program and Neural Network models using the reported load, tracing, and inference measurements. Done means identifying whether the slowdown is expected or a reproducible coremltools issue and documenting the conditions, since no repository test or source file is named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning, python
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100