Imports of large Python library (aws-cdk-lib) extremely slow
- Dominant language
- TypeScript
- Stars
- 2.9k
- Forks
- 267
- Avg merge
- 1d 25m
- Merged PRs (30d)
- 14
Description
## :bug: Bug Report
### Affected Languages
- [ ] `TypeScript` or `Javascript`
- [x] `Python`
- [ ] `Java`
- [ ] .NET (`C#`, `F#`, ...)
- [ ] `Go`
### General Information
* **JSII Version:** N/A
* **Platform:** MacOS
### What is the problem?
Originally reported as https://github.com/aws/aws-cdk/issues/19000
----------------------------------------
I noticed that simple commands such as `cdk ls` would take very long (sometimes above 20 seconds, rarely much less) to complete. This in a project created using `cdk init sample-app --language python`.
### Reproduction Steps
```shell
# setup:
$ cdk init sample-app --language python
$ source .venv/bin/activate
$ pip install -r requirements.txt
# reproducing issue:
$ time python -c "import aws_cdk as cdk"
python -c "import aws_cdk as cdk" 5.74s user 4.20s system 58% cpu 17.113 total
```
### What did you expect to happen?
A simple import should not take longer than 1 second (ideally even less, but there are constrains that make that diffucult here I understand).
### What actually happened?
As seen above, it takes ~17 seconds. This is quite consistent, however first or second time after initing the project, it can take 20-25 seconds.
### CDK CLI Version
2.12.0 (build c9786db)
### Framework Version
-
### Node.js Version
v16.14.0
### OS
MacOS 11.6.2 (20G314) - Intel Macbook
### Language
Python
### Language Version
Python (3.8.9, 3.9.10)
### Other information
To better understand where the time went, I debugged the issue with some print statements:
```
❯ time python -c "import aws_cdk as cdk"
process.py start: 0.00
self._process.args=['node', '--max-old-space-size=4069', '/var/folders/9j/cszr_w557ns8q4bk2ctzygx40000gp/T/tmp9_s6gne5/bin/jsii-runtime.js']
>>> b'{"hello":"@jsii/runtime@1.54.0"}\n'
process.py before self._next_message(): 0.25
>>> b'{"ok":{"assembly":"constructs","types":10}}\n'
process.py after self._next_message(): 0.33
process.py before self._next_message(): 0.33
>>> b'{"ok":{"assembly":"aws-cdk-lib","types":7639}}\n'
process.py after self._next_message(): 12.95
python -c "import aws_cdk as cdk" 5.72s user 4.21s system 58% cpu 16.923 total
```
The offending method seems to be (from Python's point of view) `jsii/_kernel/providers/process.py: _NodeProcess.send`
I understand that this communicates with a node process, so I assume it to be something going on there that takes very long.
Contributor guide
Research direction
Reproduce the delay with `python -c "import aws_cdk as cdk"` and inspect `jsii/_kernel/providers/process.py`, especially `_NodeProcess.send`. Trace the Node process interaction through the `constructs` and `aws-cdk-lib` assembly responses, then verify that importing `aws_cdk` completes within the reported sub-second target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, python, typescript
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100