(aws-lambda-python-alpha): Failed to install numpy 2.3.0 with Python 3.11 or lower
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
When creating a PythonFunction (using aws_lambda_python_alpha) with runtime Python 3.11, we cannot install the latest version of NumPy (2.3.0).
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Library Version
_No response_
### Expected Behavior
Numpy 2.3.0 installs successfully inside the Lambda build container.
### Current Behavior
```
$ cdk synth --app "python app.py"
#0 building with "builder" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.56kB 0.0s done
#1 DONE 0.0s
#2 [internal] load metadata for public.ecr.aws/sam/build-python3.11:latest
#2 DONE 0.0s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [1/2] FROM public.ecr.aws/sam/build-python3.11:latest
#4 DONE 0.0s
#5 [2/2] RUN python -m venv /usr/app/venv && mkdir /tmp/pip-cache && chmod -R 777 /tmp/pip-cache && pip install --upgrade pip && mkdir /tmp/poetry-cache && chmod -R 777 /tmp/poetry-cache && pip install pipenv==2022.4.8 poetry==1.5.1 && rm -rf /tmp/pip-cache/* /tmp/poetry-cache/*
#5 CACHED
#6 exporting to image
#6 exporting layers done
#6 writing image sha256:c2672077b57445177f3dd8ea27ebc99b20f9012a1f75e80fc665135ffff4d823 done
#6 naming to docker.io/library/cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 done
#6 DONE 0.0s
Bundling asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage...
sending incremental file list
app.py
requirements.txt
sent 473 bytes received 54 bytes 1,054.00 bytes/sec
total size is 291 speedup is 0.55
Collecting numpy==2.3.0 (from -r requirements.txt (line 1))
Downloading numpy-2.3.0.tar.gz (20.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 34.0 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [19 lines of output]
+ /usr/app/venv/bin/python /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/vendored-meson/meson/meson.py setup /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7/meson-python-native-file.ini
The Meson build system
Version: 1.6.1
Source dir: /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e
Build dir: /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7
Build type: native build
Project name: NumPy
Project version: 2.3.0
C compiler for the host machine: cc (gcc 7.3.1 "cc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
C linker for the host machine: cc ld.bfd 2.29.1-31
C++ compiler for the host machine: c++ (gcc 7.3.1 "c++ (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
C++ linker for the host machine: c++ ld.bfd 2.29.1-31
Cython compiler for the host machine: cython (cython 3.1.2)
Host machine cpu family: x86_64
Host machine cpu: x86_64
../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 9.3
A full log can be found at /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7/meson-logs/meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
jsii.errors.JavaScriptError:
Error: Failed to bundle asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage, bundle output is located at /project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building: Error: docker exited with status 1
--> Command: docker run --rm -u "1000:1000" -v "/project/samples/broken_numpy/example_lambda:/asset-input:delegated" -v "/project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building:/asset-output:delegated" -w "/asset-input" cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 bash -c "rsync -rLv /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output"
at AssetStaging.bundle (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:512)
at AssetStaging.stageByBundling (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:5358)
at stageThisAsset (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2728)
at Cache.obtain (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
at new AssetStaging (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:3125)
at new Asset (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:1252)
at AssetCode.bind (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:5:4375)
at new Function2 (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:11029)
at new Function2 (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/prop-injectable.js:1:488)
at new PythonFunction (/tmp/jsii-kernel-kCXuHL/node_modules/@aws-cdk/aws-lambda-python-alpha/lib/function.js:48:9)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/project/samples/broken_numpy/app.py", line 9, in
PythonFunction(
File "/project/venv/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in __call__
inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/venv/lib/python3.11/site-packages/aws_cdk/aws_lambda_python_alpha/__init__.py", line 1120, in __init__
jsii.create(self.__class__, self, [scope, id, props])
File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 334, in create
response = self.provider.create(
^^^^^^^^^^^^^^^^^^^^^
File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 365, in create
return self._process.send(request, CreateResponse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 342, in send
raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Failed to bundle asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage, bundle output is located at /project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building: Error: docker exited with status 1
--> Command: docker run --rm -u "1000:1000" -v "/project/samples/broken_numpy/example_lambda:/asset-input:delegated" -v "/project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building:/asset-output:delegated" -w "/asset-input" cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 bash -c "rsync -rLv /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output"
```
### Reproduction Steps
Create a stack with a 3.11 lambda using PythonFunction
```py
from aws_cdk import App, Stack
from aws_cdk.aws_lambda_python_alpha import PythonFunction
from aws_cdk.aws_lambda import Runtime
app = App()
stack = Stack(app, "BrokenLambdaInstallStack")
PythonFunction(
stack,
"BrokenLambdaInstall",
entry="example_lambda",
index="app.py",
runtime=Runtime.PYTHON_3_11,
)
app.synth()
```
Create the ``example_lambda`` folder with a requirements.txt with ``numpy==2.3.0``
### Possible Solution
AWS CDK uses the Dockerfile located at packages/@aws-cdk/aws-lambda-python-alpha/lib/Dockerfile to create the build environment for Lambda functions.
For Python 3.11, it uses the image public.ecr.aws/sam/build-python3.11:latest
(see: https://github.com/aws/aws-sam-build-images/blob/develop/build-image-src/Dockerfile-python311),
which is based on public.ecr.aws/lambda/python:3.11.
As mentioned in the NumPy issue [numpy/numpy#29150](https://github.com/numpy/numpy/issues/29150),
public.ecr.aws/lambda/python:3.11 is based on Amazon Linux 2, and NumPy no longer provides prebuilt wheels for it.
Simple test:
```
docker run -it --rm --entrypoint /bin/bash public.ecr.aws/lambda/python:3.11 1 ↵
bash-4.2# pip install numpy==2.3.0
Collecting numpy==2.3.0
Downloading numpy-2.3.0.tar.gz (20.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 44.1 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
+ /var/lang/bin/python3.11 /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/vendored-meson/meson/meson.py setup /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6/meson-python-native-file.ini
The Meson build system
Version: 1.6.1
Source dir: /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d
Build dir: /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6
Build type: native build
Project name: NumPy
Project version: 2.3.0
../meson.build:1:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'], ['clang'], ['nvc'], ['pgcc'], ['icc'], ['icx']]
The following exception(s) were encountered:
Running `cc --version` gave "[Errno 2] No such file or directory: 'cc'"
Running `gcc --version` gave "[Errno 2] No such file or directory: 'gcc'"
Running `clang --version` gave "[Errno 2] No such file or directory: 'clang'"
Running `nvc --version` gave "[Errno 2] No such file or directory: 'nvc'"
Running `pgcc --version` gave "[Errno 2] No such file or directory: 'pgcc'"
Running `icc --version` gave "[Errno 2] No such file or directory: 'icc'"
Running `icx --version` gave "[Errno 2] No such file or directory: 'icx'"
A full log can be found at /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6/meson-logs/meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[notice] A new release of pip is available: 24.0 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
```
### Additional Information/Context
Related issue in numpy repo https://github.com/numpy/numpy/issues/29150
### AWS CDK Library version (aws-cdk-lib)
2.200.1
### AWS CDK CLI version
2.1018.0
### Node.js Version
v22.16.01
### OS
Debian12
### Language
Python
### Language Version
3.11
### Other information
_No response_
Contributor guide
Research direction
Start with packages/@aws-cdk/aws-lambda-python-alpha/lib/Dockerfile and reproduce the failure using the Python 3.11 example with numpy==2.3.0. Compare the configured SAM build image with the Lambda base image and determine a compatible build environment; done means the reported cdk synth reproduction completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- build-system, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100