aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

Glue Crawler RecrawlPolicy doesn't work with SQS queue

Open
#1,849 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::Glue::Crawler

### Resource Name

_No response_

### Issue Description

Created a Glue crawler and an SQS queue to push S3 event notifications with associated permissions in a single template.

```typescript
const crawler = new aws_glue.CfnCrawler(this, 'analyticsCrawler', {
role: crawlerRole.roleArn,
targets: {
s3Targets: [
{
path: `s3://${this.props.dataBucket.bucketName}/`,
sampleSize: 10,
eventQueueArn: crawlerQueue.queueArn,
},
],
},
databaseName: database.databaseName,
schedule: { scheduleExpression: 'cron(15 17 ? * FRI *)' },
recrawlPolicy: { recrawlBehavior: 'CRAWL_EVENT_MODE' },
schemaChangePolicy: {
deleteBehavior: 'DEPRECATE_IN_DATABASE',
updateBehavior: 'UPDATE_IN_DATABASE',
},
});
```

### Expected Behavior

Both the Queue and Crawler should deploy and the Crawler should use the RecrawlPolicy to only crawl the relevant files.

### Observed Behavior

Error message:

```
"eventTime": "2023-11-08T16:52:48Z",
"eventSource": "glue.amazonaws.com",
"eventName": "CreateCrawler",
"awsRegion": "eu-west-1",
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "InvalidInputException",
"errorMessage": "SQS queue arn:aws:sqs:eu-west-1:: does not exist or the role provided does not have access to it.",
"requestParameters": {
"role": "arn:aws:iam:::role/",
"schedule": "cron(15 17 ? * FRI *)",
```

### Test Cases

Deploying in a single template fails
Deploying the queue first and then the crawler works fine, but deploying them at the same time fails.
Adding a dependency to the crawler on the queue also doesn't work

### Other Details

Spoken to AWS technical support (Case ID 14257963701) who said there was an internal ticket, but suggested raising a bug here too

Contributor guide

Open the contributing guide

Research direction

Start with the AWS::Glue::Crawler resource and the provided TypeScript reproduction. Compare deploying the queue and crawler together with deploying the queue first, and verify whether the reported dependency behavior is reproducible. Done means both resources deploy in one template and CRAWL_EVENT_MODE is accepted for the crawler.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.