aws / aws/sagemaker-python-sdk
CodeArtifact login support for FrameworkProcessor Jobs
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the feature you'd like**
We use private Python packages hosted in CodeArtifact for some of our common preprocessing and training code. However, currently there is no way to have a processing job authenticate to CodeArtifact before it installs Python requirements from a `requirements.txt` file.
Currently, we get an authorization token from CodeArtifact, write it to a file, and include that file with our processing/training code that gets pushed to S3 and pulled down in the jobs. This is really just a cleaner way to implement the same thing.
**How would this feature be used? Please describe.**
The simplest way to implement this would be to borrow code from the [sagemaker-training-toolkit](https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/modules.py#L186-L235) package to parse a CodeArtifact repo URL from a CodeArtifact repo ARN, and then simply write this token as an index into the `runproc.sh` file that gets generated in the [_generate_framework_script()](https://github.com/akuma12/sagemaker-python-sdk/blob/master/src/sagemaker/processing.py#L1830-L1864) method of the FrameworkProcessor class.
Adding an optional `codeartifact_repo_arn` parameter to the `.run()` method would allow it to be passed to the `_generate_framework_script()` method.
Contributor guide
Research direction
Start with src/sagemaker/processing.py, especially FrameworkProcessor.run() and _generate_framework_script(), then compare the CodeArtifact URL parsing in sagemaker-training-toolkit modules.py. Trace how runproc.sh installs requirements.txt. Done means an optional repository ARN can be passed through and the generated job authenticates to CodeArtifact before installing private packages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100