(aws-lambda-python): cache Docker layer with dependencies
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
Bundling Python Lambdas that contain `requirements.txt`, `Pipfile`, or `poetry.lock` file happens in a Docker container. Firstly, the `requirements.txt` file is generated (for pipenv and Poetry), and then dependencies are installed.
However, this happens each time any code change is made. Each time you change Lambda code (not its dependencies), the Docker build performs the above steps, downloading libraries from the internet.
This is the responsible code:
https://github.com/aws/aws-cdk/blob/4613886eda2c06f75f17768bf47b4d7f14d52e4a/packages/%40aws-cdk/aws-lambda-python/lib/bundling.ts#L109-L122
Dependencies change far less often than the code. Best practices for building in Docker are to firstly download dependencies and only then copy the code. This allows the dependencies layer to be cached, and on consecutive runs, only the code is updated while the dependencies layer is cached.
### Use Case
This will greatly reduce consecutive Lambda bundling times, as dependencies will be fetched from the internet only when they change. When only the code changes, a cached Docker layer with dependencies will be used.
### Proposed Solution
In short, bundling Python Lambda should be changed from:
1. Copy all source files (line 113)
2. Generate `requirements.txt` (line 115)
3. Install dependencies (line 117)
to:
1. Copy dependencies files (i.e. `poetry.lock`)
2. Generate `requirements.txt`
3. Install dependencies
4. Copy the rest of the files
### Other Information
I am willing to implement it after greenlighting by the CDK team.
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.61.1
### Environment details (OS name and version, etc.)
any
Contributor guide
Research direction
Start with packages/@aws-cdk/aws-lambda-python/lib/bundling.ts around lines 109-122, where the Docker bundling steps are assembled. Trace how requirements.txt is generated for requirements.txt, Pipfile, and poetry.lock inputs. Done means dependency installation can use a cached Docker layer when only Lambda source changes, while dependency changes trigger installation again.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python, typescript
- Domain
- build-system, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100