aws / aws/aws-cdk

(glue): add a property for `--additional-python-modules` option

Open
#21,753 0 comments 8 reactions 0 assignees View on GitHub
@aws-cdk/aws-glue effort/small feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

Glue job v2 or later supports to use pip to install additional modules by specifying `--additional-python-modules` option. [doc](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#addl-python-modules-support)
e.g.

```sh
# specify module like [module_name]==[version]
--additional-python-modules scikit-learn==0.21.3,ulid==1.1.0

# you can also install a wheel from s3 bucket
--additional-python-modules s3://aws-glue-native-spark/tests/j4.2/ephem-3.7.7.1-cp37-cp37m-linux_x86_64.whl
```

It will be convenient if we have a handy API for this, just like `--extra-files` or `--extra-py-files`, which are [already supported](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.PythonShellExecutableProps.html#extrafilesspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan).

### Use Case

Currently we need to specify this option in [`defaultArguments`](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Job.html#defaultargumentsspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan) property, which is difficult to memorize and use. A dedicated api for this feature will help lots of developers.

### Proposed Solution

Add a property like `additionalPythonModules` to `JobExecutableProps`. We need to abstract python module so that we can both specify modules from pypi and modules from s3.

API will be like:

```ts
interface PythonExecutableProps {
...
// add this
readonly additionalPythonModules?: IPythonModule[];
// will be bound as
// --additional-python-modules: additionalPythonModules?.map(m=>m.toString()).join(',')
}

interface IPythonModule {
toString: ()=>string;
}

class PyPiModule implements IPythonModule {
constructor(name: string, version: string) {}
toString() { return `${name}==${version}` }
}

class S3Module implements IPythonModule {
constructor(bucket: IBucket, key: string) {}
toString() { return bucket.s3UrlForObject(key) }
}
```

### Other Information

_No response_

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.37.1

### Environment details (OS name and version, etc.)

macOS

Contributor guide

Open the contributing guide

Research direction

Start by locating JobExecutableProps and the existing extra-files and extra-py-files APIs, then trace how defaultArguments are bound to Glue job arguments. Compare the proposed IPythonModule, PyPiModule, and S3Module shapes with the repository's conventions. Done means a dedicated property supports both PyPI specifications and S3 wheel URLs and produces the --additional-python-modules argument.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.