aws / aws/sagemaker-python-sdk

Add support for multiple requirements.txt files

Open
#2,690 1 comment 0 reactions 1 assignee Claimed by @nargokul View on GitHub
component: pysdk-team type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**

Right now, its expected that `requirements.txt` will be found (**With that name**) in the script directory. In some cases, multiple requirements.txt files are needed to install the correct dependencies. Unfortunately, right now you'd have to edit your code in order to merge to one unified requirements.txt file if you have multiple.

**How would this feature be used? Please describe.**

Allow `dependencies` kwarg to pick up any `.txt` files as requirements files when they're added to the list.

```python
estimator = HuggingFace(
...
dependencies=['./requirements.txt', '../_test_requirements.txt']
)
```

**Describe alternatives you've considered**
Right now, I'm:
- merging all requirements files a user specifies in a CLI tool I made
- Writing the merged requirements to a new tmp directory
- Adding all python files to the tmp directory as well
- Launching training runs out of the temp directory

I've also seen folks installing requirements directly in their python files, but that just feels wrong to me.

**Additional context**
...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.