@aws-cdk/aws-lambda-python-alpha: PIP_INDEX_URL in buildArgs is not applied
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Even if `PIP_INDEX_URL` is specified in buildArgs, this is not valid when applying requirements.txt. The reason is that the `PIP_INDEX_URL` specified in the Dockerfile is only specified in the `ARG`, so this environment variable is no longer valid when `pip install -r requirements.txt` is executed (when the container is run).
This problem can be solved by specifying `PIP_INDEX_URL` in the `environment` in addition to buildArgs, but this method is redundant.
```typescript
const pipIndexUrl = spawnSync("pip", ["config", "get", "global.index-url"]).stdout.toString().trim();
const bundling = pipIndexUrl == "" ? undefined : {
buildArgs: { PIP_INDEX_URL: pipIndexUrl },
environment: { PIP_INDEX_URL: pipIndexUrl },
};
const lambdaLayer = new PythonLayerVersion(this, "LambdaLayer", {
layerVersionName: "SomethingLayer",
entry: "/lambda/layer",
compatibleRuntimes: [lambda.Runtime.PYTHON_3_9],
bundling,
});
```
Adding similar values such as `PIP_EXTRA_INDEX_URL`, `HTTPS_PROXY` and `POETRY_VERSION` further accentuates the redundancy.
Another possible method is to put `--index-url` in the requirements.txt file, but this is not desirable because it is necessary to change the description in the file each time the environment is known.
### Expected Behavior
The specification of `buildArgs` must also affect the installation of libraries in the requirements.txt file.
### Current Behavior
In an environment where `PIP_INDEX_URL` needs to be changed, if I only change `PIP_INDEX_URL` in `buildArgs`, CDK execution will stop without access to the pypi.org site for installation of the libraries defined in requirements.txt.
Specifically, pypi.org connection timeouts and attempts are repeated.
```
Step 11/11 : CMD [ "python" ]
---> Running in 6792f94ec7f7
Removing intermediate container 6792f94ec7f7
---> da1c164310a2
Successfully built da1c164310a2
Successfully tagged cdk-025e0dd8bb67991ff31bd34edeaca2bd4c2201094770be7454fdbf71d6e13bf4:latest
Bundling asset MyStack/LambdaLayer/Code/Stage...
sending incremental file list
created directory /asset-output/python
./
requirements.txt
sent 120 bytes received 81 bytes 402.00 bytes/sec
total size is 13 speedup is 0.06
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pillow/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pillow/
```
### Reproduction Steps
1. Prepare a closed environment that cannot access pypi.org directly (ex. an EC2 instance in a private subnet with no Internet gateway)
2. Prepare a separate Python package index other than pipy.org (ex. CodeArtifact)
3. Prepare a Lambda (Python) function with requirements.txt containing optional libraries
4. Run the following CDK stack in a closed environment (A separately prepared package index should be accessible)
```typescript
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import {
BundlingOptions,
PythonFunction,
PythonLayerVersion,
} from "@aws-cdk/aws-lambda-python-alpha";
import { spawnSync } from "child_process";
export class MyLambdaStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const pipIndexUrl = spawnSync("pip", ["config", "get", "global.index-url"])
.stdout.toString()
.trim();
const bundling: BundlingOptions | undefined =
pipIndexUrl != ""
? { buildArgs: { PIP_INDEX_URL: pipIndexUrl } }
: undefined;
const lambdaFunction = new PythonFunction(this, "function", {
entry: "/path/to/function",
runtime: lambda.Runtime.PYTHON_3_9,
bundling,
});
}
}
```
### Possible Solution
Several proposals exist to correct this problem.
1. Modify the Dockerfile that aws-lambda-python-alpha has
```Dockerfile
# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE
ARG PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
ARG POETRY_VERSION=1.5.1
```
to
```Dockerfile
# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE
ARG PIP_INDEX_URL
ENV PIP_INDEX_URL=$PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ENV PIP_EXTRA_INDEX_URL=$PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
ENV HTTPS_PROXY=$HTTPS_PROXY
ARG POETRY_VERSION=1.5.1
ENV POETRY_VERSION=$POETRY_VERSION
```
2. In terms of preventing redundant code, one possible approach would be to use only the environment without buildArgs. In that case, you need to modify the DockerImage fromBuild to use environment.
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/core/lib/bundling.ts#L354-L363
3. Allow passing environment variables to bundlingCommand. Add PIP_INDEX_URL to the environment variable at the point where the command is generated.
https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda-python-alpha/lib/bundling.ts#L119-L134
### Additional Information/Context
_No response_
### CDK CLI Version
2.92.0
### Framework Version
_No response_
### Node.js Version
16.20.2
### OS
Linux
### Language
Typescript
### Language Version
TypeScript (3.7.5)
### Other information
_No response_
Contributor guide
Research direction
Start by reading packages/@aws-cdk/aws-lambda-python-alpha/lib/bundling.ts around lines 119-134 and core/lib/bundling.ts around lines 354-363, then inspect how buildArgs and environment reach the Docker build and requirements installation. Done means a PIP_INDEX_URL supplied through buildArgs is available when requirements.txt is installed without requiring a duplicate environment setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python, typescript
- Domain
- build-system, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100