aws / aws/sagemaker-python-sdk
sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)
- Langage dominant
- Python
- Étoiles
- 2.3k
- Forks
- 1.3k
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 35
Description
## PySDK Version
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
Reported against the `sagemaker-core` distribution, version 2.15.0 (repo tag `v3.15.0`).
## Describe the bug
`sagemaker-core` 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. `ModelTrainer(...)` → `TrainDefaults.get_role`) before any training job is submitted: `resolve_and_validate_role` → `_evaluate_permissions` → `iam:SimulatePrincipalPolicy`. It raises `RoleValidationError` on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (`OrganizationsDecisionDetail.AllowedByOrganizations`).
Per [AWS docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html), the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, `SimulatePrincipalPolicy` returns `AllowedByOrganizations: false` (with `EvalDecision: implicitDeny`, `MatchedStatements: []`) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.
Two observable consequences:
1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
2. The same role works fine from a notebook / via a direct `create_training_job` call, because those paths don't run this client-side pre-check.
2.15.0 is currently the [latest published release](https://pypi.org/project/sagemaker-core/), so there is no fixed version to upgrade to.
## To reproduce
Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts `sagemaker.amazonaws.com` and grants the training smoke-test actions at `Resource: *`; a calling identity that can call `iam:SimulatePrincipalPolicy`.
```bash
pip install 'sagemaker-core==2.15.0'
```
```python
from sagemaker.core.helper.iam_role_resolver import IamRoleResolver
# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
role_arn="arn:aws:iam:::role/",
role_type="training",
)
```
**Result:**
```
RoleValidationError: IAM role 'arn:aws:iam:::role/' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer
```
Confirm the verdict is an org-layer artifact rather than a real permission gap:
```bash
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam:::role/ \
--action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob
```
```json
{
"EvalActionName": "cloudwatch:PutMetricData",
"EvalDecision": "implicitDeny",
"MatchedStatements": [],
"OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
// ...same for the other actions, including sagemaker:CreateTrainingJob —
// yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}
```
The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.
## Expected behavior
The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an `AllowedByOrganizations: false` result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. `validate_role=False` or an env var) would also let users bypass the client-side check without modifying IAM.
## Screenshots or logs
```
.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570 # Permission check (definitive denial blocks; unverifiable warns)
571 verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572 if verdict is False:
> 573 raise RoleValidationError(
574 _build_validation_error_message(role_arn, role_type, miss...
```
## System information
- **SageMaker Python SDK version:** sagemaker-core 2.15.0 (repo tag `v3.15.0`)
- **Framework name or algorithm:** N/A — fails during role validation, before framework/job selection (framework-agnostic)
- **Framework version:** N/A
- **Python version:** 3.12
- **CPU or GPU:** N/A (fails before job submission)
- **Custom Docker image (Y/N):** N/A
## Additional context
Introduced in `sagemaker-core` 2.15.0 — file `sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py`, added in commit `dba1127a` ("New release (#5969)"), first tag `v3.15.0`. Authoring PRs: #2041 (added `SimulatePrincipalPolicy`-based `resolve_or_create_role`) → #2080 (replaced it with the raising `resolve_and_validate_role`). #2080 notes it gates only on `*`-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.
**Suggested fix:** in `_evaluate_permissions`, when an action is `implicitDeny` with no matched identity/SCP statement and `OrganizationsDecisionDetail.AllowedByOrganizations == false`, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.
**Relevant docs:**
- [IAM policy simulator can't test SCPs with conditions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html#:~:text=The%20policy%20simulator%20doesn%27t%20evaluate%20SCPs%20that%20have%20any%20conditions.)
- [IAM policy evaluation logic (explicit Deny overrides Allow)](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html)
**Workarounds (both verified):**
1. Attach an explicit Deny on `iam:SimulatePrincipalPolicy` to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
2. Pin `sagemaker-core<2.15.0`, which predates the pre-check.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, en particulier resolve_and_validate_role et _evaluate_permissions, puis reproduisez la réponse du simulateur IAM à l’aide de la commande fournie aws iam simulate-principal-policy. Le travail est considéré comme terminé lorsque les résultats de SCP fondés sur des conditions ne produisent plus de RoleValidationError erroné, que les refus d’autorisation réels continuent d’être traités correctement et que le comportement existant de warn-and-proceed est préservé.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, python
- Domaine
- cloud, security
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 64/100