aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
Glue Crawler RecrawlPolicy doesn't work with SQS queue
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
### Name of the resource
AWS::Glue::Crawler
### Resource Name
_No response_
### Issue Description
Created a Glue crawler and an SQS queue to push S3 event notifications with associated permissions in a single template.
```typescript
const crawler = new aws_glue.CfnCrawler(this, 'analyticsCrawler', {
role: crawlerRole.roleArn,
targets: {
s3Targets: [
{
path: `s3://${this.props.dataBucket.bucketName}/`,
sampleSize: 10,
eventQueueArn: crawlerQueue.queueArn,
},
],
},
databaseName: database.databaseName,
schedule: { scheduleExpression: 'cron(15 17 ? * FRI *)' },
recrawlPolicy: { recrawlBehavior: 'CRAWL_EVENT_MODE' },
schemaChangePolicy: {
deleteBehavior: 'DEPRECATE_IN_DATABASE',
updateBehavior: 'UPDATE_IN_DATABASE',
},
});
```
### Expected Behavior
Both the Queue and Crawler should deploy and the Crawler should use the RecrawlPolicy to only crawl the relevant files.
### Observed Behavior
Error message:
```
"eventTime": "2023-11-08T16:52:48Z",
"eventSource": "glue.amazonaws.com",
"eventName": "CreateCrawler",
"awsRegion": "eu-west-1",
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "InvalidInputException",
"errorMessage": "SQS queue arn:aws:sqs:eu-west-1:: does not exist or the role provided does not have access to it.",
"requestParameters": {
"role": "arn:aws:iam:::role/",
"schedule": "cron(15 17 ? * FRI *)",
```
### Test Cases
Deploying in a single template fails
Deploying the queue first and then the crawler works fine, but deploying them at the same time fails.
Adding a dependency to the crawler on the queue also doesn't work
### Other Details
Spoken to AWS technical support (Case ID 14257963701) who said there was an internal ticket, but suggested raising a bug here too
Contributor guide
Research direction
Start with the AWS::Glue::Crawler resource and the provided TypeScript reproduction. Compare deploying the queue and crawler together with deploying the queue first, and verify whether the reported dependency behavior is reproducible. Done means both resources deploy in one template and CRAWL_EVENT_MODE is accepted for the crawler.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100