aws / aws/aws-cdk

step-functions: AWS Glue Workflows

Open
#29,001 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-stepfunctions effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

Currently, we can make use of [CallAwsService](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_stepfunctions_tasks.CallAwsService.html), however it would be nice to have a dedicated Construct in place for the Glue Workflow to be used also.

### Use Case

If we want to merge both workflows with step functions, this seems to make sense. Glue workflows can very well be easily reproduced to step functions, however some developers may prefer to use them, when wanting to use step functions to manage an overall data pipeline, and use workflows to handle the smaller ETL pipelines.

My specific use case is to trigger a large state machine that will handle multiple smaller glue workflows, reducing cost whilst still maintaining the original data pipeline.

### Proposed Solution

Similar to Glue, we could add another construct `GlueStartWorkflowRun`.

```ts
new tasks.GlueStartWorkflowRun(this, 'Task', {
glueWorkflowName: 'my-glue-job',
... //other parameters
});
```

### Other Information

A quick query, more than anything. How are synchronous jobs triggered? For example, glue `StartJobRun` includes a `.sync` option, however I was wondering how that is reproduced within the CDK? The only reason I'm asking is the need for whether or not I should include that functionality also in a possible PR if this was to be worked on.

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.126.0

### Environment details (OS name and version, etc.)

Windows 11

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Glue-related construct and the linked CallAwsService documentation to understand the expected AWS Step Functions task shape. Define the GlueStartWorkflowRun inputs, including whether synchronous execution is required, and verify the construct can trigger a Glue Workflow as described in the proposed usage. Done means the dedicated construct supports the agreed workflow-start behavior and has coverage for its parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.