NeuralNetwork and MLprogram lead to very different results

## ❓Question

Hello! I'm using coremltools to convert a custom architecture from pytorch to coreML. I've noticed that I only get correct results if I use the neuralnetwork backend and not the mlprogram backend. I'm wondering if anyone has any high(or low) level insights as to why this might be the case? And where I might be able to start tracing the differences in representations to find the supposed bug? Assuming this is a bug

The issue names no files, tests, or reproducible model. Start by reproducing the custom PyTorch conversion with both the neuralnetwork and mlprogram backends, then compare their generated representations. Done means isolating a confirmed conversion discrepancy and documenting a minimal reproduction.

Written by the indexing model from the issue text.