Azure / Azure/azure-cli

Improve embedded python speed

Open
#25,106 1 comment 0 reactions 1 assignee Claimed by @jiasli View on GitHub
act-platform-engineering-squad Auto-Assign Azure CLI Team feature-request Packaging
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Is your feature request related to a problem? Please describe.**

CLI executes with embedded Python interpreter on Windows.

Compare with my dev environment, each command is noticeably slower.
For example: `Measure-Command {az account -h}` takes **763**ms and **342**ms in MSI and dev env respectively.

### Uncompress python3.10.zip?
**I think uncompressing zip makes python run faster, but I can't get a stable result to compare the difference. [PEP273](https://peps.python.org/pep-0273/) provide a 20-year-old benchmark, does the modern hardware widen the gap?**

After some research, I find a difference between standard Python and embedded Python:
> The standard library is included as pre-compiled and optimized .pyc files in a ZIP -- [The embeddable package](https://docs.python.org/3/using/windows.html#the-embeddable-package)

If build-in packages is a zip file, Python need to uncompress it before import statement. This makes sense.
To verify this, if `C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip` is remove, run `python.exe` raises this:
```
╰─ ./python.exe ─╯
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
sys.base_prefix = ''
sys.base_exec_prefix = ''
sys.platlibdir = 'lib'
sys.executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
sys.prefix = ''
sys.exec_prefix = ''
sys.path = [
'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip',
'.\\DLLs',
'.\\lib',
'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
```
We can extract `python3.10.zip` to `python3.10.zip`/`Lib` **folder** to save the uncompression time.
In standard Python, there files are in `Lib` folder, we can also uncompress it there.
After this operation, `Measure-Command {az account -h}` takes **684**ms, which is reduced by **10%**.
It is still slower than dev env. I'll keep working on it.

### Use 64bit embedded python?

In my dev env, I use 64bit Python, so I also try install CLI on 64bit embedded version. It's about **100**ms faster than 32bit.
But migration is not easy and it's a breaking change. https://github.com/Azure/azure-cli/pull/24607

We've released the 64-bit package in 2.51.

### Convert packages to zip package?

Some packages consist of many files. Compression it to a zip file may speed up the load time.

----

There is another way to uncompress `python3.10.zip`, but it's too complex. I'd record it here.
Create a `python310._pth` in CLI folder with this content and put its content into `python310` folder. (ref:[_pth files - Python](https://docs.python.org/3/library/sys_path_init.html#pth-files)). Please mind that after add `python310._pth`, the current folder is not in `sys.path` in embedded Python. (https://github.com/python/cpython/issues/77192)
```
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python310
import site
```
PS: Add `name.pth` file in `sys.prefix` does not work because `site` package is not found when `python310.zip` is missing ([site — Site-specific configuration hook](https://docs.python.org/3/library/site.html)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.