aws / aws/sagemaker-python-sdk

CodeArtifact login support for FrameworkProcessor Jobs

已关闭
#4,144 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
component: training type: feature request
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**Describe the feature you'd like**
We use private Python packages hosted in CodeArtifact for some of our common preprocessing and training code. However, currently there is no way to have a processing job authenticate to CodeArtifact before it installs Python requirements from a `requirements.txt` file.

Currently, we get an authorization token from CodeArtifact, write it to a file, and include that file with our processing/training code that gets pushed to S3 and pulled down in the jobs. This is really just a cleaner way to implement the same thing.

**How would this feature be used? Please describe.**
The simplest way to implement this would be to borrow code from the [sagemaker-training-toolkit](https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/modules.py#L186-L235) package to parse a CodeArtifact repo URL from a CodeArtifact repo ARN, and then simply write this token as an index into the `runproc.sh` file that gets generated in the [_generate_framework_script()](https://github.com/akuma12/sagemaker-python-sdk/blob/master/src/sagemaker/processing.py#L1830-L1864) method of the FrameworkProcessor class.

Adding an optional `codeartifact_repo_arn` parameter to the `.run()` method would allow it to be passed to the `_generate_framework_script()` method.

贡献指南

打开贡献指南

调研方向

从 src/sagemaker/processing.py 开始,重点查看 FrameworkProcessor.run() 和 _generate_framework_script(),然后对比 sagemaker-training-toolkit modules.py 中的 CodeArtifact URL 解析。跟踪 runproc.sh 如何安装 requirements.txt。完成的标准是可以传递一个可选的 repository ARN,并且生成的 job 会在安装私有 package 之前向 CodeArtifact 进行身份验证。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
cloud, machine-learning
Issue 类型
功能
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
38/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。