aws / aws/aws-cdk

Glue: Creation of partition indexes fails if the table and account permissions are managed with Lakeformation

Open
#21,415 3 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-glue bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the bug

After this merge was made:

https://github.com/aws/aws-cdk/pull/17998

We got the ability to add partition indexes, however, if I have a data lake in my account and manage the permissions via Lake Formation, I get the following error on deployment in cloud trail:

```
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXX",
"arn": "XXXX",
"accountId": "XXXXX",
"accessKeyId": "XXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXX",
"arn": "XXXXX",
"accountId": "XXXXXX",
"userName": "XXXXXX"
},
"attributes": {
"creationDate": "2022-08-02T02:17:14Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2022-08-02T02:19:01Z",
"eventSource": "glue.amazonaws.com",
"eventName": "CreatePartitionIndex",
"awsRegion": "us-east-1",
"sourceIPAddress": "XXXXX",
"userAgent": "aws-sdk-nodejs/2.1186.0 linux/v14.19.3 exec-env/AWS_Lambda_nodejs14.x promise",
"errorCode": "AccessDenied",
"errorMessage": "An unknown error occurred",
"requestParameters": {
"databaseName": "dbname",
"tableName": "tablename",
"partitionIndex": {
"keys": [
"my_col"
],
"indexName": "indexname"
}
},
"responseElements": null,
"requestID": "XXXX",
"eventID": "XXXX",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXX",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"clientProvidedHostHeader": "glue.us-east-1.amazonaws.com"
}
}
```

And I can see this in the console in the deleted stack:

```
Received response status [FAILED] from custom resource. Message returned: Insufficient Lake Formation permission(s): Required Alter on tablename (RequestId: XXXX)
```

From what i can tell the custom resource created by CDK for this operation is deleted upon fail, so I can't see the policy it has. Also not sure how to grant permissions in advance if this doesn't exist.

### Expected Behavior

A role/policy can be provided to the table construct so that we can provide LF permissions in advance.

### Current Behavior

The custom resource is created with a policy we have no control over, therefore it is not possible to add index partitions to a data lake managed by LF.

### Reproduction Steps

It's tricky to add these steps as a whole data lake would have to be created for this.

### Possible Solution

A role/policy can be provided to the table construct so that we can provide LF permissions in advance.

### Additional Information/Context

_No response_

### CDK CLI Version

1.163.2 (build 1a28d63)

### Framework Version

_No response_

### Node.js Version

N/A

### OS

N/A

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Glue table construct's partition-index custom resource and its CreatePartitionIndex call, then review how the resource policy is currently created. Determine how a caller could provide the role or policy needed for Lake Formation Alter permission on the table. Done means the table construct exposes that control and partition-index creation works with permissions granted in advance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
authorization, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.