[RFC] Canonicalize python file structure

🚀 The feature, motivation and pitch

Blocking #2871

Context

Currently the file structure looks like this:

executorch
├── backends
...
├── exir
├── extension
...
├── runtime
├── setup.py
...
└── version.txt

Problem

• Since we don't have an executorch/ directory, we can't add code into executorch root module, we will have to do things like import executorch.exir etc. In the future we want to do from executorch import <top level API>
• We want to have a clean separation between python code and C++ code, for easier build system management.
  • E.g., we don't want to build python code in GH, so it will be easier to configure the internal repo to not map TARGETS files to GH.
• Editable mode install couldn't find the top level module correctly and thus failed to install.
  • This is because when setuptools tries to map the module namespace to actual files, it will do the following mapping: 'executorch/backens.apple': ['<path to local executorch repo>/executorch/backends/apple']. setuptools couldn't define a root level executorch module as well because it only lookup folders under executorch/. See NAMESPACE section in this generated code, instead we want this code.

Proposal

Move all python code into executorch/ and use that as our root level module. Keep all native source files intact.

executorch
├── backends
│   ├── apple // no python code
│   ├── arm
├── executorch // only python code
│   ├── backends
│   ├── exir
│   └── extension
...
└── version.txt

This way we can build a self-contained executorch module which works for editable mode.

Migrations

If we go down this route, we will have to migrate python targets to executorch/ and change internal usage. I don't expect this to affect pip users since they will just do the same from executorch.exir import X.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response