aws / aws/sagemaker-python-sdk

sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

Open
#6,019 2 comments 3 reactions 0 assignees View on GitHub
component: pysdk-team type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

## PySDK Version

- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

Reported against the `sagemaker-core` distribution, version 2.15.0 (repo tag `v3.15.0`).

## Describe the bug

`sagemaker-core` 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. `ModelTrainer(...)` → `TrainDefaults.get_role`) before any training job is submitted: `resolve_and_validate_role` → `_evaluate_permissions` → `iam:SimulatePrincipalPolicy`. It raises `RoleValidationError` on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (`OrganizationsDecisionDetail.AllowedByOrganizations`).

Per [AWS docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html), the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, `SimulatePrincipalPolicy` returns `AllowedByOrganizations: false` (with `EvalDecision: implicitDeny`, `MatchedStatements: []`) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.

Two observable consequences:

1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
2. The same role works fine from a notebook / via a direct `create_training_job` call, because those paths don't run this client-side pre-check.

2.15.0 is currently the [latest published release](https://pypi.org/project/sagemaker-core/), so there is no fixed version to upgrade to.

## To reproduce

Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts `sagemaker.amazonaws.com` and grants the training smoke-test actions at `Resource: *`; a calling identity that can call `iam:SimulatePrincipalPolicy`.

```bash
pip install 'sagemaker-core==2.15.0'
```

```python
from sagemaker.core.helper.iam_role_resolver import IamRoleResolver

# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
role_arn="arn:aws:iam:::role/",
role_type="training",
)
```

**Result:**

```
RoleValidationError: IAM role 'arn:aws:iam:::role/' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer
```

Confirm the verdict is an org-layer artifact rather than a real permission gap:

```bash
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam:::role/ \
--action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob
```

```json
{
"EvalActionName": "cloudwatch:PutMetricData",
"EvalDecision": "implicitDeny",
"MatchedStatements": [],
"OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
// ...same for the other actions, including sagemaker:CreateTrainingJob —
// yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}
```

The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.

## Expected behavior

The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an `AllowedByOrganizations: false` result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. `validate_role=False` or an env var) would also let users bypass the client-side check without modifying IAM.

## Screenshots or logs

```
.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570 # Permission check (definitive denial blocks; unverifiable warns)
571 verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572 if verdict is False:
> 573 raise RoleValidationError(
574 _build_validation_error_message(role_arn, role_type, miss...
```

## System information

- **SageMaker Python SDK version:** sagemaker-core 2.15.0 (repo tag `v3.15.0`)
- **Framework name or algorithm:** N/A — fails during role validation, before framework/job selection (framework-agnostic)
- **Framework version:** N/A
- **Python version:** 3.12
- **CPU or GPU:** N/A (fails before job submission)
- **Custom Docker image (Y/N):** N/A

## Additional context

Introduced in `sagemaker-core` 2.15.0 — file `sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py`, added in commit `dba1127a` ("New release (#5969)"), first tag `v3.15.0`. Authoring PRs: #2041 (added `SimulatePrincipalPolicy`-based `resolve_or_create_role`) → #2080 (replaced it with the raising `resolve_and_validate_role`). #2080 notes it gates only on `*`-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.

**Suggested fix:** in `_evaluate_permissions`, when an action is `implicitDeny` with no matched identity/SCP statement and `OrganizationsDecisionDetail.AllowedByOrganizations == false`, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.

**Relevant docs:**

- [IAM policy simulator can't test SCPs with conditions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html#:~:text=The%20policy%20simulator%20doesn%27t%20evaluate%20SCPs%20that%20have%20any%20conditions.)
- [IAM policy evaluation logic (explicit Deny overrides Allow)](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html)

**Workarounds (both verified):**

1. Attach an explicit Deny on `iam:SimulatePrincipalPolicy` to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
2. Pin `sagemaker-core<2.15.0`, which predates the pre-check.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.