Improve embedded python speed
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Is your feature request related to a problem? Please describe.**
CLI executes with embedded Python interpreter on Windows.
Compare with my dev environment, each command is noticeably slower.
For example: `Measure-Command {az account -h}` takes **763**ms and **342**ms in MSI and dev env respectively.
### Uncompress python3.10.zip?
**I think uncompressing zip makes python run faster, but I can't get a stable result to compare the difference. [PEP273](https://peps.python.org/pep-0273/) provide a 20-year-old benchmark, does the modern hardware widen the gap?**
After some research, I find a difference between standard Python and embedded Python:
> The standard library is included as pre-compiled and optimized .pyc files in a ZIP -- [The embeddable package](https://docs.python.org/3/using/windows.html#the-embeddable-package)
If build-in packages is a zip file, Python need to uncompress it before import statement. This makes sense.
To verify this, if `C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip` is remove, run `python.exe` raises this:
```
╰─ ./python.exe ─╯
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
sys.base_prefix = ''
sys.base_exec_prefix = ''
sys.platlibdir = 'lib'
sys.executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
sys.prefix = ''
sys.exec_prefix = ''
sys.path = [
'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip',
'.\\DLLs',
'.\\lib',
'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
```
We can extract `python3.10.zip` to `python3.10.zip`/`Lib` **folder** to save the uncompression time.
In standard Python, there files are in `Lib` folder, we can also uncompress it there.
After this operation, `Measure-Command {az account -h}` takes **684**ms, which is reduced by **10%**.
It is still slower than dev env. I'll keep working on it.
### Use 64bit embedded python?
In my dev env, I use 64bit Python, so I also try install CLI on 64bit embedded version. It's about **100**ms faster than 32bit.
But migration is not easy and it's a breaking change. https://github.com/Azure/azure-cli/pull/24607
We've released the 64-bit package in 2.51.
### Convert packages to zip package?
Some packages consist of many files. Compression it to a zip file may speed up the load time.
----
There is another way to uncompress `python3.10.zip`, but it's too complex. I'd record it here.
Create a `python310._pth` in CLI folder with this content and put its content into `python310` folder. (ref:[_pth files - Python](https://docs.python.org/3/library/sys_path_init.html#pth-files)). Please mind that after add `python310._pth`, the current folder is not in `sys.path` in embedded Python. (https://github.com/python/cpython/issues/77192)
```
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python310
import site
```
PS: Add `name.pth` file in `sys.prefix` does not work because `site` package is not found when `python310.zip` is missing ([site — Site-specific configuration hook](https://docs.python.org/3/library/site.html)).
Contributor guide
Assessment
This issue has not been assessed yet.