flyteorg / flyteorg/flyte

[Connector] Expand AWS SageMaker managed job support

Open
#7,767 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

## Summary

Expand `flytekitplugins-awssagemaker` beyond model deployment and endpoint management with first-class async connectors for SageMaker managed jobs.

## Proposed scope

- Training jobs
- Processing jobs
- Batch Transform jobs
- Hyperparameter Tuning jobs
- Inference Recommender jobs

Each task follows Flyte's async connector contract:

- `create()` submits the corresponding SageMaker job
- `get()` polls the describe API and maps SageMaker states to Flyte phases
- `delete()` stops the job idempotently
- terminal responses are projected into stable, downstream-friendly result dictionaries

The implementation reuses the plugin's existing boto3 configuration templating (`{inputs.X}`, `{images.X}`, and `{idempotence_token}`), requires no FlytePropeller changes, and registers each connector through the existing `flytekit.plugins` entry-point mechanism.

## Follow-up

A separate follow-up will add Flyte-native Pythonic execution for Training and Processing, where a normal `@task` function body runs inside the SageMaker container through `ContainerEntrypoint`. Keeping that execution model separate allows the conventional boto3/config connector surface to be reviewed and merged independently.

## Validation

- Full SageMaker plugin unit suite
- Standard flytekit pre-commit checks (`ruff`, formatting, codespell, pydoclint)
- Connector registration/import validation
- AWS smoke validation for the managed job lifecycle

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing model-deployment and endpoint-management connectors in flytekitplugins-awssagemaker, then inspect the flytekit.plugins entry-point registration and the plugin unit suite. Done means all five managed job types support create(), get(), and delete(), stable terminal results, registration validation, passing pre-commit checks, and successful AWS lifecycle smoke validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.