aws / aws/sagemaker-python-sdk

SageMakerClient ignores the passed boto3 session for the "sagemaker" client since 2.13.0 — all resource API calls sign with default-chain credentials

Open
#5,986 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**TL;DR** — Since sagemaker-core 2.13.0, `SageMakerClient` ignores the `session=` the caller passed and builds its main `sagemaker` client from a fresh default-chain session. Every V3 resource API call (`FeatureGroup.create`, `Model.create`, `describe`/`delete`/`list`, …) is silently signed with ambient credentials instead of the caller's.

Besides breaking explicit-session workflows, this is a quiet identity mix-up with a security edge: code written to run under a scoped or assumed role can instead execute under a less-privileged (or worse a more-privileged) ambient identity, with no error or warning. Last good version: 2.12.0. Workaround: pin `sagemaker-core>=2.12,<2.13`.

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
`SageMakerClient.__init__` (`sagemaker-core/src/sagemaker/core/utils/utils.py`) contains a block introduced by #5919 (merged 2026-06-03, released as sagemaker-core 2.13.0) that loads a custom service model for pre-GA Job APIs through a brand-new botocore session:

```python
# TODO: Remove post-launch. This loads a custom botocore service model
# (from the 'sample/' directory) that includes pre-GA Job APIs ...
bc_session = bc_session_mod.get_session()
...
custom_session = Session(botocore_session=bc_session, region_name=env_region)

self.sagemaker_client = custom_session.client("sagemaker", ...) # <-- passed `session` ignored
```

`custom_session` resolves credentials from the default provider chain (env vars / `AWS_PROFILE` / IMDS), discarding the passed session's credentials. The other clients built in the same constructor (`sagemaker-runtime`, `sagemaker-featurestore-runtime`, `sagemaker-metrics`) still correctly use the passed `session`.

Since every resource class routes through `Base.get_sagemaker_client(session=...)` → `SageMakerClient(...).get_client("sagemaker")`, all V3 resource API calls are affected.

**To reproduce**
Minimal proof, no AWS resources created — the returned client does not hold the passed session's credentials:

```python
import boto3
from sagemaker.core.utils.utils import SageMakerClient

# Any session whose credentials differ from the default provider chain,
# e.g. an assumed role:
sts = boto3.client("sts")
creds = sts.assume_role(
RoleArn="arn:aws:iam:::role/",
RoleSessionName="repro",
)["Credentials"]
session = boto3.Session(
aws_access_key_id=creds["AccessKeyId"],
aws_secret_access_key=creds["SecretAccessKey"],
aws_session_token=creds["SessionToken"],
region_name="us-west-2",
)

client = SageMakerClient(session=session, region_name="us-west-2").get_client("sagemaker")
client_key = client._request_signer._credentials.get_frozen_credentials().access_key

print("passed session key:", session.get_credentials().access_key)
print("client signs with :", client_key)
assert client_key == session.get_credentials().access_key, "client ignored the passed session"
```

The assert passes on 2.12.0 and fails on 2.13.0–2.15.0.

The same code with sagemaker-core 2.12.0 (identical credentials, identical account) succeeds.

**Expected behavior**
When a `session` is passed to `SageMakerClient` (directly or via any resource method's `session=` parameter), every client it constructs — including `sagemaker` — signs requests with that session's credentials. If the custom service-model loader is still needed, attach it to the *passed* session's botocore session rather than creating a new default-chain session.

**System information**
- **SageMaker Python SDK version**: sagemaker 3.13.1 / sagemaker-core 2.13.1 (broken); still present in sagemaker-core 2.15.0; last good 2.12.0
- **Framework name / algorithm**: n/a (Feature Store / core resource classes)
- **Python version**: 3.13
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N (bare venv on macOS)

**Additional context**
- The bug is easy to miss on EC2/ECS/Batch/SageMaker jobs because the default chain resolves to the attached role — the wrong session happens to be the right identity. It bites exactly when an explicit session matters: assumed roles, cross-account work, multi-profile laptops.
- Two smaller quirks in the same block, worth fixing together:
- `SageMakerClient` is a singleton (`SingletonMeta`), so even the correctly-handled clients only honor whichever session arrives first in the process.
- `logger.info(f"Runs on sagemaker {env_stage}, region:{env_region}")` logs on every construction and reads like leftover debug output.
- Introduced by #5919

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.