aws / aws/aws-cdk

@aws-cdk/aws-lambda-python-alpha: PIP_INDEX_URL in buildArgs is not applied

Open
#26,920 5 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-lambda @aws-cdk/aws-lambda-python bug p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Even if `PIP_INDEX_URL` is specified in buildArgs, this is not valid when applying requirements.txt. The reason is that the `PIP_INDEX_URL` specified in the Dockerfile is only specified in the `ARG`, so this environment variable is no longer valid when `pip install -r requirements.txt` is executed (when the container is run).

This problem can be solved by specifying `PIP_INDEX_URL` in the `environment` in addition to buildArgs, but this method is redundant.

```typescript
const pipIndexUrl = spawnSync("pip", ["config", "get", "global.index-url"]).stdout.toString().trim();
const bundling = pipIndexUrl == "" ? undefined : {
buildArgs: { PIP_INDEX_URL: pipIndexUrl },
environment: { PIP_INDEX_URL: pipIndexUrl },
};

const lambdaLayer = new PythonLayerVersion(this, "LambdaLayer", {
layerVersionName: "SomethingLayer",
entry: "/lambda/layer",
compatibleRuntimes: [lambda.Runtime.PYTHON_3_9],
bundling,
});
```
Adding similar values such as `PIP_EXTRA_INDEX_URL`, `HTTPS_PROXY` and `POETRY_VERSION` further accentuates the redundancy.

Another possible method is to put `--index-url` in the requirements.txt file, but this is not desirable because it is necessary to change the description in the file each time the environment is known.

### Expected Behavior

The specification of `buildArgs` must also affect the installation of libraries in the requirements.txt file.

### Current Behavior

In an environment where `PIP_INDEX_URL` needs to be changed, if I only change `PIP_INDEX_URL` in `buildArgs`, CDK execution will stop without access to the pypi.org site for installation of the libraries defined in requirements.txt.
Specifically, pypi.org connection timeouts and attempts are repeated.
```
Step 11/11 : CMD [ "python" ]
---> Running in 6792f94ec7f7
Removing intermediate container 6792f94ec7f7
---> da1c164310a2
Successfully built da1c164310a2
Successfully tagged cdk-025e0dd8bb67991ff31bd34edeaca2bd4c2201094770be7454fdbf71d6e13bf4:latest
Bundling asset MyStack/LambdaLayer/Code/Stage...
sending incremental file list
created directory /asset-output/python
./
requirements.txt

sent 120 bytes received 81 bytes 402.00 bytes/sec
total size is 13 speedup is 0.06
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pillow/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pillow/
```

### Reproduction Steps

1. Prepare a closed environment that cannot access pypi.org directly (ex. an EC2 instance in a private subnet with no Internet gateway)
2. Prepare a separate Python package index other than pipy.org (ex. CodeArtifact)
3. Prepare a Lambda (Python) function with requirements.txt containing optional libraries
4. Run the following CDK stack in a closed environment (A separately prepared package index should be accessible)

```typescript
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import {
BundlingOptions,
PythonFunction,
PythonLayerVersion,
} from "@aws-cdk/aws-lambda-python-alpha";
import { spawnSync } from "child_process";

export class MyLambdaStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

const pipIndexUrl = spawnSync("pip", ["config", "get", "global.index-url"])
.stdout.toString()
.trim();
const bundling: BundlingOptions | undefined =
pipIndexUrl != ""
? { buildArgs: { PIP_INDEX_URL: pipIndexUrl } }
: undefined;

const lambdaFunction = new PythonFunction(this, "function", {
entry: "/path/to/function",
runtime: lambda.Runtime.PYTHON_3_9,
bundling,
});
}
}
```

### Possible Solution

Several proposals exist to correct this problem.

1. Modify the Dockerfile that aws-lambda-python-alpha has

```Dockerfile
# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE

ARG PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
ARG POETRY_VERSION=1.5.1
```

to

```Dockerfile
# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE

ARG PIP_INDEX_URL
ENV PIP_INDEX_URL=$PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ENV PIP_EXTRA_INDEX_URL=$PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
ENV HTTPS_PROXY=$HTTPS_PROXY
ARG POETRY_VERSION=1.5.1
ENV POETRY_VERSION=$POETRY_VERSION
```

2. In terms of preventing redundant code, one possible approach would be to use only the environment without buildArgs. In that case, you need to modify the DockerImage fromBuild to use environment.
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/core/lib/bundling.ts#L354-L363

3. Allow passing environment variables to bundlingCommand. Add PIP_INDEX_URL to the environment variable at the point where the command is generated.
https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-lambda-python-alpha/lib/bundling.ts#L119-L134

### Additional Information/Context

_No response_

### CDK CLI Version

2.92.0

### Framework Version

_No response_

### Node.js Version

16.20.2

### OS

Linux

### Language

Typescript

### Language Version

TypeScript (3.7.5)

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading packages/@aws-cdk/aws-lambda-python-alpha/lib/bundling.ts around lines 119-134 and core/lib/bundling.ts around lines 354-363, then inspect how buildArgs and environment reach the Docker build and requirements installation. Done means a PIP_INDEX_URL supplied through buildArgs is available when requirements.txt is installed without requiring a duplicate environment setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, typescript
Domain
build-system, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.