aws / aws/sagemaker-python-sdk
CodeArtifact login support for FrameworkProcessor Jobs
- Vorherrschende Sprache
- Python
- Sterne
- 2.3k
- Forks
- 1.3k
- Ø Merge
- 1 T. 22 Std.
- Gemergte PRs (30 T.)
- 35
Beschreibung
**Describe the feature you'd like**
We use private Python packages hosted in CodeArtifact for some of our common preprocessing and training code. However, currently there is no way to have a processing job authenticate to CodeArtifact before it installs Python requirements from a `requirements.txt` file.
Currently, we get an authorization token from CodeArtifact, write it to a file, and include that file with our processing/training code that gets pushed to S3 and pulled down in the jobs. This is really just a cleaner way to implement the same thing.
**How would this feature be used? Please describe.**
The simplest way to implement this would be to borrow code from the [sagemaker-training-toolkit](https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/modules.py#L186-L235) package to parse a CodeArtifact repo URL from a CodeArtifact repo ARN, and then simply write this token as an index into the `runproc.sh` file that gets generated in the [_generate_framework_script()](https://github.com/akuma12/sagemaker-python-sdk/blob/master/src/sagemaker/processing.py#L1830-L1864) method of the FrameworkProcessor class.
Adding an optional `codeartifact_repo_arn` parameter to the `.run()` method would allow it to be passed to the `_generate_framework_script()` method.
Beitragsleitfaden
Rechercherichtung
Beginne mit src/sagemaker/processing.py, insbesondere FrameworkProcessor.run() und _generate_framework_script(), und vergleiche anschließend das Parsen der CodeArtifact-URL in sagemaker-training-toolkit modules.py. Verfolge, wie runproc.sh requirements.txt installiert. Erledigt ist die Aufgabe, wenn ein optionaler Repository-ARN durchgereicht werden kann und der generierte Job sich vor der Installation privater Pakete bei CodeArtifact authentifiziert.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- aws, python
- Bereich
- cloud, machine-learning
- Issue-Typ
- Feature
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 38/100