aws / aws/sagemaker-python-sdk
SageMakerClient ignores the passed boto3 session for the "sagemaker" client since 2.13.0 — all resource API calls sign with default-chain credentials
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**TL;DR** — Since sagemaker-core 2.13.0, `SageMakerClient` ignores the `session=` the caller passed and builds its main `sagemaker` client from a fresh default-chain session. Every V3 resource API call (`FeatureGroup.create`, `Model.create`, `describe`/`delete`/`list`, …) is silently signed with ambient credentials instead of the caller's.
Besides breaking explicit-session workflows, this is a quiet identity mix-up with a security edge: code written to run under a scoped or assumed role can instead execute under a less-privileged (or worse a more-privileged) ambient identity, with no error or warning. Last good version: 2.12.0. Workaround: pin `sagemaker-core>=2.12,<2.13`.
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
`SageMakerClient.__init__` (`sagemaker-core/src/sagemaker/core/utils/utils.py`) contains a block introduced by #5919 (merged 2026-06-03, released as sagemaker-core 2.13.0) that loads a custom service model for pre-GA Job APIs through a brand-new botocore session:
```python
# TODO: Remove post-launch. This loads a custom botocore service model
# (from the 'sample/' directory) that includes pre-GA Job APIs ...
bc_session = bc_session_mod.get_session()
...
custom_session = Session(botocore_session=bc_session, region_name=env_region)
self.sagemaker_client = custom_session.client("sagemaker", ...) # <-- passed `session` ignored
```
`custom_session` resolves credentials from the default provider chain (env vars / `AWS_PROFILE` / IMDS), discarding the passed session's credentials. The other clients built in the same constructor (`sagemaker-runtime`, `sagemaker-featurestore-runtime`, `sagemaker-metrics`) still correctly use the passed `session`.
Since every resource class routes through `Base.get_sagemaker_client(session=...)` → `SageMakerClient(...).get_client("sagemaker")`, all V3 resource API calls are affected.
**To reproduce**
Minimal proof, no AWS resources created — the returned client does not hold the passed session's credentials:
```python
import boto3
from sagemaker.core.utils.utils import SageMakerClient
# Any session whose credentials differ from the default provider chain,
# e.g. an assumed role:
sts = boto3.client("sts")
creds = sts.assume_role(
RoleArn="arn:aws:iam:::role/",
RoleSessionName="repro",
)["Credentials"]
session = boto3.Session(
aws_access_key_id=creds["AccessKeyId"],
aws_secret_access_key=creds["SecretAccessKey"],
aws_session_token=creds["SessionToken"],
region_name="us-west-2",
)
client = SageMakerClient(session=session, region_name="us-west-2").get_client("sagemaker")
client_key = client._request_signer._credentials.get_frozen_credentials().access_key
print("passed session key:", session.get_credentials().access_key)
print("client signs with :", client_key)
assert client_key == session.get_credentials().access_key, "client ignored the passed session"
```
The assert passes on 2.12.0 and fails on 2.13.0–2.15.0.
The same code with sagemaker-core 2.12.0 (identical credentials, identical account) succeeds.
**Expected behavior**
When a `session` is passed to `SageMakerClient` (directly or via any resource method's `session=` parameter), every client it constructs — including `sagemaker` — signs requests with that session's credentials. If the custom service-model loader is still needed, attach it to the *passed* session's botocore session rather than creating a new default-chain session.
**System information**
- **SageMaker Python SDK version**: sagemaker 3.13.1 / sagemaker-core 2.13.1 (broken); still present in sagemaker-core 2.15.0; last good 2.12.0
- **Framework name / algorithm**: n/a (Feature Store / core resource classes)
- **Python version**: 3.13
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N (bare venv on macOS)
**Additional context**
- The bug is easy to miss on EC2/ECS/Batch/SageMaker jobs because the default chain resolves to the attached role — the wrong session happens to be the right identity. It bites exactly when an explicit session matters: assumed roles, cross-account work, multi-profile laptops.
- Two smaller quirks in the same block, worth fixing together:
- `SageMakerClient` is a singleton (`SingletonMeta`), so even the correctly-handled clients only honor whichever session arrives first in the process.
- `logger.info(f"Runs on sagemaker {env_stage}, region:{env_region}")` logs on every construction and reads like leftover debug output.
- Introduced by #5919
Contributor guide
Assessment
This issue has not been assessed yet.