glue-alpha: feedback on spark jobs constructs introduced in "Refactored glue-alpha L2 CDK construct RFC 0497"
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 74
Description
### Describe the feature
Hello CDK team,
As a user of `glue-alpha`, and having contributed the initial job construct in https://github.com/aws/aws-cdk/pull/12506, I recently came across the refactor from https://github.com/aws/aws-cdk/pull/32521 as part of updating my CDK applications to `2.178.0`.
Having refactored my code to leverage the new constructs - mostly the `ScalaSparkEtlJob` - I noticed a couple of things and wanted to provide feedback on them
## Job Role requirements and its auto-creation
- In the previous version, providing a `role` as part of `JobProps` was optional, and its absence led to the auto-creation of a default role
- In the new base `JobProperties` (sidenote: should it be called `JobProps` instead of `JobProperties`?), [role](https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-glue-alpha/lib/jobs/job.ts#L325) is now required
- Moreover, code in subclasses seem to be wrongly doing something I don't understand e.g.
- https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts#L89
- https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts#L89
I think the previous behaviour of making `role` prop optional is a sensible default, and that behaviour should be restored, and code in subclasses corrected.
## Enums / Constants
- Several classes were changed to become enums e.g. `GlueVersion` and `WorkerType` which removed the ability to use newer values easily as was possible before e.g. `GlueVersion.of(...)`.
- Documentation on those new enums still reference no-longer existing methods of `GlueVersion.of` and `WorkerType.of` e.g. https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-glue-alpha/lib/constants.ts#L5
I think the previous approach, was better, and inline with other constructs e.g. Lambda, and that it should be restored to provide an easy way for adopting new values for `GlueVersion` and `WorkerType` without the need to use escape hatches or be blocked on CDK updates.
## `extraJars`, `extraFiles`, `extraPythonFiles`, `extraJarsFirst`
- the following points are for spark jobs of different languages
- `extraJars`, `extraJarsFirst` and `extraFiles` are applicable to all spark jobs regardless of language (`Scala`/`Python`)
- `extraJars` allow spark to load jvm-based libraries that can be used across both `Scala` and `Python` spark jobs
- `extraJarsFirst` is about the order of jar loading for all spark jobs across both `Scala` and `Python`
- `extraFiles` is a way to load other files e.g. binary files or text files in spark - again regardless of the spark job's language
- `extraPythonFiles` is only relevant to `Python` spark jobs
- the new constructs are not implementing the above behaviour
- `extraJars` is not implemented for python spark jobs
- `extraJarsFirst` is implemented in only `ScalaSparkFlexEtlJob` even though it should be available in all spark jobs wherever `extraJars` is available
- `extraFiles` seem to be completely missing from `Scala` spark jobs
Therefore
- all spark jobs should be updated to support `extraJars`, `extraJarsFirst`, `extraFiles` props (I found a related https://github.com/aws/aws-cdk/pull/33238)
### Use Case
N/A
### Proposed Solution
N/A
### Other Information
N/A
### Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.178.0
### Environment details (OS name and version, etc.)
macOS
Contributor guide
Research direction
Start with packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts, scala-spark-etl-job.ts, and constants.ts, then review the related pull request 33238 and the referenced construct refactor. Compare the previous and current APIs for role creation, enum constants, and Spark job file props. Done means the requested behavior is restored consistently across Spark job constructs and the affected documentation matches the API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100