step-functions: AWS Glue Workflows
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
Currently, we can make use of [CallAwsService](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_stepfunctions_tasks.CallAwsService.html), however it would be nice to have a dedicated Construct in place for the Glue Workflow to be used also.
### Use Case
If we want to merge both workflows with step functions, this seems to make sense. Glue workflows can very well be easily reproduced to step functions, however some developers may prefer to use them, when wanting to use step functions to manage an overall data pipeline, and use workflows to handle the smaller ETL pipelines.
My specific use case is to trigger a large state machine that will handle multiple smaller glue workflows, reducing cost whilst still maintaining the original data pipeline.
### Proposed Solution
Similar to Glue, we could add another construct `GlueStartWorkflowRun`.
```ts
new tasks.GlueStartWorkflowRun(this, 'Task', {
glueWorkflowName: 'my-glue-job',
... //other parameters
});
```
### Other Information
A quick query, more than anything. How are synchronous jobs triggered? For example, glue `StartJobRun` includes a `.sync` option, however I was wondering how that is reproduced within the CDK? The only reason I'm asking is the need for whether or not I should include that functionality also in a possible PR if this was to be worked on.
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.126.0
### Environment details (OS name and version, etc.)
Windows 11
Contributor guide
Research direction
Start by reviewing the existing Glue-related construct and the linked CallAwsService documentation to understand the expected AWS Step Functions task shape. Define the GlueStartWorkflowRun inputs, including whether synchronous execution is required, and verify the construct can trigger a Glue Workflow as described in the proposed usage. Done means the dedicated construct supports the agreed workflow-start behavior and has coverage for its parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100