(glue): add a property for `--additional-python-modules` option
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
Glue job v2 or later supports to use pip to install additional modules by specifying `--additional-python-modules` option. [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#addl-python-modules-support)
e.g.
```sh
# specify module like [module_name]==[version]
--additional-python-modules scikit-learn==0.21.3,ulid==1.1.0
# you can also install a wheel from s3 bucket
--additional-python-modules s3://aws-glue-native-spark/tests/j4.2/ephem-3.7.7.1-cp37-cp37m-linux_x86_64.whl
```
It will be convenient if we have a handy API for this, just like `--extra-files` or `--extra-py-files`, which are [already supported](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.PythonShellExecutableProps.html#extrafilesspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan).
### Use Case
Currently we need to specify this option in [`defaultArguments`](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Job.html#defaultargumentsspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan) property, which is difficult to memorize and use. A dedicated api for this feature will help lots of developers.
### Proposed Solution
Add a property like `additionalPythonModules` to `JobExecutableProps`. We need to abstract python module so that we can both specify modules from pypi and modules from s3.
API will be like:
```ts
interface PythonExecutableProps {
...
// add this
readonly additionalPythonModules?: IPythonModule[];
// will be bound as
// --additional-python-modules: additionalPythonModules?.map(m=>m.toString()).join(',')
}
interface IPythonModule {
toString: ()=>string;
}
class PyPiModule implements IPythonModule {
constructor(name: string, version: string) {}
toString() { return `${name}==${version}` }
}
class S3Module implements IPythonModule {
constructor(bucket: IBucket, key: string) {}
toString() { return bucket.s3UrlForObject(key) }
}
```
### Other Information
_No response_
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.37.1
### Environment details (OS name and version, etc.)
macOS
Contributor guide
Research direction
Start by locating JobExecutableProps and the existing extra-files and extra-py-files APIs, then trace how defaultArguments are bound to Glue job arguments. Compare the proposed IPythonModule, PyPiModule, and S3Module shapes with the repository's conventions. Done means a dedicated property supports both PyPI specifications and S3 wheel URLs and produces the --additional-python-modules argument.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100