aws / aws/aws-cdk

(aws-lambda-python): cache Docker layer with dependencies

Open
#23,829 23 comments 36 reactions 0 assignees View on GitHub
@aws-cdk/aws-lambda-python feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

Bundling Python Lambdas that contain `requirements.txt`, `Pipfile`, or `poetry.lock` file happens in a Docker container. Firstly, the `requirements.txt` file is generated (for pipenv and Poetry), and then dependencies are installed.

However, this happens each time any code change is made. Each time you change Lambda code (not its dependencies), the Docker build performs the above steps, downloading libraries from the internet.

This is the responsible code:

https://github.com/aws/aws-cdk/blob/4613886eda2c06f75f17768bf47b4d7f14d52e4a/packages/%40aws-cdk/aws-lambda-python/lib/bundling.ts#L109-L122

Dependencies change far less often than the code. Best practices for building in Docker are to firstly download dependencies and only then copy the code. This allows the dependencies layer to be cached, and on consecutive runs, only the code is updated while the dependencies layer is cached.

### Use Case

This will greatly reduce consecutive Lambda bundling times, as dependencies will be fetched from the internet only when they change. When only the code changes, a cached Docker layer with dependencies will be used.

### Proposed Solution

In short, bundling Python Lambda should be changed from:

1. Copy all source files (line 113)
2. Generate `requirements.txt` (line 115)
3. Install dependencies (line 117)

to:

1. Copy dependencies files (i.e. `poetry.lock`)
2. Generate `requirements.txt`
3. Install dependencies
4. Copy the rest of the files

### Other Information

I am willing to implement it after greenlighting by the CDK team.

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.61.1

### Environment details (OS name and version, etc.)

any

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/aws-lambda-python/lib/bundling.ts around lines 109-122, where the Docker bundling steps are assembled. Trace how requirements.txt is generated for requirements.txt, Pipfile, and poetry.lock inputs. Done means dependency installation can use a cached Docker layer when only Lambda source changes, while dependency changes trigger installation again.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, typescript
Domain
build-system, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.