aws / aws/aws-cdk

[glue] Support Glue::Crawler as L2 construct

Open
#8,863 9 comments 22 reactions 0 assignees View on GitHub
@aws-cdk/aws-glue effort/large feature-request p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

We are getting more heavily invested into AWS Glue. Right now we have some L2 constructs internally and it would be good to have them upstream.

### Use Case

Have L2 Construct for L1 `CfnCrawler`.

### Proposed Solution

Have a workflow like this:

```typescript
const bucket = new s3.Bucket(stack, 'bucket');
new glue.Crawler(stack, 'crawler', {
targets: [
new S3Target(bucket, {exclusions: ["abc"]})
]
});
```

**Here is a test, that we run for additional info**:

```typescript
test('Create Crawler', () => {
// GIVEN
const app = new cdk.App();
const stack = new cdk.Stack(app, 'stack');
const bucket = new s3.Bucket(stack, 'bucket', {bucketName: "test"});

// WHEN
new glue.Crawler(stack, 'crawler', {
targets: [new S3Target(bucket, {exclusions: ["abc"]})]
});

// THEN
cdkExpect(stack).to(haveResource('AWS::Glue::Crawler',
{
Type: 'AWS::Glue::Crawler',
Properties: {
Role: { "Fn::GetAtt": ["crawlerServiceRole63A6FAF9", "Arn"]},
Targets:{
S3Targets: [{
Path: { "Fn::Join": [ "", [ "s3://", {Ref: "bucket43879C71"}]]},
Exclusions: ["abc"]
}]
},
Configuration: "{\"Version\":1,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}},\"Grouping\":{\"TableGroupingPolicy\":\"CombineCompatibleSchemas\"}}"
}
}, ResourcePart.CompleteDefinition));

cdkExpect(stack).to(haveResourceLike('AWS::IAM::Role',
{
Type: 'AWS::IAM::Role',
Properties: {
AssumeRolePolicyDocument: {
Statement: [
{
Action: "sts:AssumeRole",
Effect: "Allow",
Principal: {Service: "glue.amazonaws.com"}
}
]},
ManagedPolicyArns: [
{
"Fn::Join": [
"", [
"arn:",
{Ref: "AWS::Partition"},
":iam::aws:policy/service-role/AWSGlueServiceRole"
]
]
}
],
}
}, ResourcePart.CompleteDefinition));

cdkExpect(stack).to(haveResourceLike('AWS::IAM::Policy',
{ PolicyDocument: {
Statement: [
{
Resource: [{
"Fn::GetAtt": ["bucket43879C71", "Arn"]
}]
}
]},
Roles: [
{ Ref: "crawlerServiceRole63A6FAF9" }
]
}
));
});
```

### Other

* [X] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start by reading the L1 CfnCrawler API and the TypeScript example and test included in the issue. The work is done when a glue.Crawler and S3Target construct can produce the shown AWS::Glue::Crawler resource, service role, and IAM policy assertions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.