aws-samples / aws-samples/sagemaker-101-workshop
Can't upgrade torchtext with newer PyTorch
- Dominant language
- Jupyter Notebook
- Stars
- 80
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
Per the [torchtext README](https://github.com/pytorch/text#installation), our current pinned torchtext version (0.6) is a long way out of sync with our PyTorch version (PTv1.8=TTv0.9, PTv1.10=TTv0.11).
I explored pinning the PT version to current and allowing pip to solve, with a statement like this:
```
!pip install torch==`pip show torch | grep 'Version:' | sed 's/Version: //'` torchtext
```
On the SMStudio PyTorch v1.10 CPU kernel, this installs the expected version of torchtext (0.11), but `import torchtext` fails due to missing symbols. Perhaps due to something missing from the CPU-optimized version of PyTorch?
So for now torchtext remains pinned at a pretty old version. We only use it for basic English text tokenization (util `tokenize_and_pad_docs()`), so maybe can switch to some other solution if this can't be resolved.
Contributor guide
Research direction
Read the torchtext README and inspect the utility tokenize_and_pad_docs(). Reproduce the torch 1.10/torchtext 0.11 installation and import failure in the SMStudio PyTorch CPU kernel, then determine whether a compatible dependency or tokenizer replacement is practical. Done means tokenization works without the old torchtext pin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100