aws / aws/jsii

Imports of large Python library (aws-cdk-lib) extremely slow

Open
#3,389 40 comments 17 reactions 0 assignees View on GitHub
language/python module/kernel p1
Dominant language
TypeScript
Stars
2.9k
Forks
267
Avg merge
1d 25m
Merged PRs (30d)
14

Description

## :bug: Bug Report

### Affected Languages
- [ ] `TypeScript` or `Javascript`
- [x] `Python`
- [ ] `Java`
- [ ] .NET (`C#`, `F#`, ...)
- [ ] `Go`

### General Information
* **JSII Version:** N/A
* **Platform:** MacOS

### What is the problem?

Originally reported as https://github.com/aws/aws-cdk/issues/19000

----------------------------------------

I noticed that simple commands such as `cdk ls` would take very long (sometimes above 20 seconds, rarely much less) to complete. This in a project created using `cdk init sample-app --language python`.

### Reproduction Steps

```shell
# setup:
$ cdk init sample-app --language python
$ source .venv/bin/activate
$ pip install -r requirements.txt
# reproducing issue:
$ time python -c "import aws_cdk as cdk"
python -c "import aws_cdk as cdk" 5.74s user 4.20s system 58% cpu 17.113 total
```

### What did you expect to happen?

A simple import should not take longer than 1 second (ideally even less, but there are constrains that make that diffucult here I understand).

### What actually happened?

As seen above, it takes ~17 seconds. This is quite consistent, however first or second time after initing the project, it can take 20-25 seconds.

### CDK CLI Version

2.12.0 (build c9786db)

### Framework Version

-

### Node.js Version

v16.14.0

### OS

MacOS 11.6.2 (20G314) - Intel Macbook

### Language

Python

### Language Version

Python (3.8.9, 3.9.10)

### Other information

To better understand where the time went, I debugged the issue with some print statements:

```
❯ time python -c "import aws_cdk as cdk"
process.py start: 0.00
self._process.args=['node', '--max-old-space-size=4069', '/var/folders/9j/cszr_w557ns8q4bk2ctzygx40000gp/T/tmp9_s6gne5/bin/jsii-runtime.js']
>>> b'{"hello":"@jsii/runtime@1.54.0"}\n'
process.py before self._next_message(): 0.25
>>> b'{"ok":{"assembly":"constructs","types":10}}\n'
process.py after self._next_message(): 0.33
process.py before self._next_message(): 0.33
>>> b'{"ok":{"assembly":"aws-cdk-lib","types":7639}}\n'
process.py after self._next_message(): 12.95
python -c "import aws_cdk as cdk" 5.72s user 4.21s system 58% cpu 16.923 total
```

The offending method seems to be (from Python's point of view) `jsii/_kernel/providers/process.py: _NodeProcess.send`

I understand that this communicates with a node process, so I assume it to be something going on there that takes very long.

Contributor guide

Open the contributing guide

Research direction

Reproduce the delay with `python -c "import aws_cdk as cdk"` and inspect `jsii/_kernel/providers/process.py`, especially `_NodeProcess.send`. Trace the Node process interaction through the `constructs` and `aws-cdk-lib` assembly responses, then verify that importing `aws_cdk` completes within the reported sub-second target.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, python, typescript
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.