tensorflow / tensorflow/datasets
Subword Text Encoder Deprecated
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
What I need help with / What I was wondering
Recently the text module of tfds has been deprecated. This library includes the subword text encoder class. According to the release notes users should switch to TF.text, but tf.text does not appear to have an equivalent encoder. Where can I find an equivalent subword text encoder that functions similar to the one recently deprecated in tfds?
What I've tried so far
I have looked at tf.text and it appears to have a full word encoder, an ngram encoder, and other encoders but none of these appear equivalent to the deprecated subword text encoder
It would be nice if...
It would be nice if there was an equivalent library that I could use.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the deprecated TFDS text module and TF.text but does not identify a repository file or test. Start by reviewing the TFDS deprecation notes and the available TF.text encoders, then determine whether an equivalent belongs in this project; done would require a decided scope and a documented or implemented replacement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100