Glue: Creation of partition indexes fails if the table and account permissions are managed with Lakeformation
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 74
Description
### Describe the bug
After this merge was made:
https://github.com/aws/aws-cdk/pull/17998
We got the ability to add partition indexes, however, if I have a data lake in my account and manage the permissions via Lake Formation, I get the following error on deployment in cloud trail:
```
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXX",
"arn": "XXXX",
"accountId": "XXXXX",
"accessKeyId": "XXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXX",
"arn": "XXXXX",
"accountId": "XXXXXX",
"userName": "XXXXXX"
},
"attributes": {
"creationDate": "2022-08-02T02:17:14Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2022-08-02T02:19:01Z",
"eventSource": "glue.amazonaws.com",
"eventName": "CreatePartitionIndex",
"awsRegion": "us-east-1",
"sourceIPAddress": "XXXXX",
"userAgent": "aws-sdk-nodejs/2.1186.0 linux/v14.19.3 exec-env/AWS_Lambda_nodejs14.x promise",
"errorCode": "AccessDenied",
"errorMessage": "An unknown error occurred",
"requestParameters": {
"databaseName": "dbname",
"tableName": "tablename",
"partitionIndex": {
"keys": [
"my_col"
],
"indexName": "indexname"
}
},
"responseElements": null,
"requestID": "XXXX",
"eventID": "XXXX",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXX",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"clientProvidedHostHeader": "glue.us-east-1.amazonaws.com"
}
}
```
And I can see this in the console in the deleted stack:
```
Received response status [FAILED] from custom resource. Message returned: Insufficient Lake Formation permission(s): Required Alter on tablename (RequestId: XXXX)
```
From what i can tell the custom resource created by CDK for this operation is deleted upon fail, so I can't see the policy it has. Also not sure how to grant permissions in advance if this doesn't exist.
### Expected Behavior
A role/policy can be provided to the table construct so that we can provide LF permissions in advance.
### Current Behavior
The custom resource is created with a policy we have no control over, therefore it is not possible to add index partitions to a data lake managed by LF.
### Reproduction Steps
It's tricky to add these steps as a whole data lake would have to be created for this.
### Possible Solution
A role/policy can be provided to the table construct so that we can provide LF permissions in advance.
### Additional Information/Context
_No response_
### CDK CLI Version
1.163.2 (build 1a28d63)
### Framework Version
_No response_
### Node.js Version
N/A
### OS
N/A
### Language
Typescript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start by tracing the Glue table construct's partition-index custom resource and its CreatePartitionIndex call, then review how the resource policy is currently created. Determine how a caller could provide the role or policy needed for Lake Formation Alter permission on the table. Done means the table construct exposes that control and partition-index creation works with permissions granted in advance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- authorization, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100