aws-redshift: Granting privileges to multiple users on a single table can cause concurrency errors
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 74
Description
### Describe the bug
With Redshift, concurrency errors are possible when running multiple `GRANT` statements for different users on the same table at the same time; for example, `GRANT SELECT ON Table TO User1` and `GRANT SELECT ON Table TO User2`. However, the Redshift `Table`/`User` CDK constructs do not account for this. During CloudFormation deployment, multiple `TablePrivileges` resources can create/update at the same time, which can cause concurrency errors and stack update failures.
### Expected Behavior
When creating/updating multiple users with privileges for common tables, the CloudFormation stack update always succeeds with no errors.
### Current Behavior
When creating/updating multiple users with privileges for common tables, the CloudFormation stack update sometimes fails, with the following error for a `TablePrivileges` resource:
```
Received response status [FAILED] from custom resource. Message returned: Statement status was FAILED: ERROR: could not complete because of conflict with concurrent transaction
Logs: /aws/lambda/MyStack-QueryRedshiftDatabase3de-0JlACCPDaXyQ at waitForStatementComplete (/var/task/util.js:34:15)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async Object.executeStatement (/var/task/util.js:18:5)
at async Promise.all (index 5) at async grantPrivileges (/var/task/privileges.js:34:5)
at async updatePrivileges (/var/task/privileges.js:52:9)
at async handler (/var/task/privileges.js:18:29) (RequestId: XXX)
```
This is sensitive to the exact times that the concurrent custom resource Lambdas execute the statements. So the failure does not always. The failure is more likely to happen the more users and tables are involved. Reproducing this was very inconsistent on our end. We got 10 failures in a row when doing deploying to our production account, but struggled to reproduce it in our test accounts. We have 6 tables and 2 users.
### Reproduction Steps
```ts
const users = Array.from(Array(10).keys()).map(n => new User(this, `User${n}`, {
adminUser: clusterAdminUserSecret,
cluster: redshiftCluster,
databaseName: databaseName,
encryptionKey: kmsKey,
username: `User${n}`,
}));
const tables = Array.from(Array(10).keys()).map(n => new Table(this, `Table${n}`, {
cluster: redshiftCluster,
adminUser: clusterAdminUserSecret,
databaseName: databaseName,
tableName: `Table${n}`,
tableColumns: [{name: 'field', dataType: 'int'}],
}));
for (const user of users) {
for (const table of tables) {
table.grant(user, TableAction.SELECT, TableAction.INSERT);
}
}
```
### Possible Solution
We worked around this on our end by introducing an arbitrary dependency between the `User`s, forcing them to be updated sequentially rather than concurrently. Something similar could potentially be done internally. In case this is difficult to fix, the documentation can be updated to clearly call out this known issue and suggest this workaround to the user.
```
user2.node.addDependency(user1);
```
### Additional Information/Context
_No response_
### CDK CLI Version
1.125.0 (build 67b4921)
### Framework Version
_No response_
### Node.js Version
12.x
### OS
Amazon Linux 2
### Language
Typescript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Trace the Redshift Table and User constructs to the TablePrivileges custom resource, then inspect the privilege.js grantPrivileges/updatePrivileges flow and util.js waitForStatementComplete path mentioned in the failure. Reproduce the concurrent GRANT scenario with the provided TypeScript example and determine whether deployment completes reliably without concurrent-transaction failures; otherwise document the user dependency workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgresql, typescript
- Domain
- cloud, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100