aws-samples / aws-samples/sagemaker-101-workshop

Can't upgrade torchtext with newer PyTorch

Open
#29 0 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Jupyter Notebook
Stars
80
Forks
51
PR merge metrics
No merged PRs in 30d

Description

Per the [torchtext README](https://github.com/pytorch/text#installation), our current pinned torchtext version (0.6) is a long way out of sync with our PyTorch version (PTv1.8=TTv0.9, PTv1.10=TTv0.11).

I explored pinning the PT version to current and allowing pip to solve, with a statement like this:

```
!pip install torch==`pip show torch | grep 'Version:' | sed 's/Version: //'` torchtext
```

On the SMStudio PyTorch v1.10 CPU kernel, this installs the expected version of torchtext (0.11), but `import torchtext` fails due to missing symbols. Perhaps due to something missing from the CPU-optimized version of PyTorch?

So for now torchtext remains pinned at a pretty old version. We only use it for basic English text tokenization (util `tokenize_and_pad_docs()`), so maybe can switch to some other solution if this can't be resolved.

Contributor guide

Open the contributing guide

Research direction

Read the torchtext README and inspect the utility tokenize_and_pad_docs(). Reproduce the torch 1.10/torchtext 0.11 installation and import failure in the SMStudio PyTorch CPU kernel, then determine whether a compatible dependency or tokenizer replacement is practical. Done means tokenization works without the old torchtext pin.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.