NVIDIA / NVIDIA/NeMo-text-processing
en TN: plural unit abbreviations are not verbalized (`"2-3 mins"` → `"two - three mins"` )
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 506
- Forks
- 186
- Avg merge
- 6d 18h
- Merged PRs (30d)
- 7
Description
Describe the bug
In English deterministic text normalization, plural written unit abbreviations (mins, hrs, secs) are not recognized as units, so measure normalization never fires on them:
"5 mins"→"five mins"(unit left unverbalized)"2-3 mins"→"two - three mins"(the measure range rule can't fire either, so the hyphen is kept — for TTS this renders as silence/a pause)
The singular abbreviations work, which isolates the cause to the unit list: data/measure/unit.tsv contains min, sec, hr — and even the colloquial plural lbs → pound — but no mins/hrs/secs.
Steps/Code to reproduce bug
from nemo_text_processing.text_normalization.normalize import Normalizer
n = Normalizer(input_case="cased", lang="en", deterministic=True)
for text in ["5 min", "5 mins", "2-3 min", "2-3 mins", "2-3 hrs", "2-3 secs"]:
print(f"{text!r:12} -> {n.normalize(text)!r}")
'5 min' -> 'five minutes'
'5 mins' -> 'five mins' # plural written form: not a unit
'2-3 min' -> 'two to three minutes'
'2-3 mins' -> 'two - three mins' # measure range rule never fires
'2-3 hrs' -> 'two - three HRS'
'2-3 secs' -> 'two - three secs'
Expected behavior
'5 mins' -> 'five minutes'
'2-3 mins' -> 'two to three minutes'
'2-3 hrs' -> 'two to three hours'
'2-3 secs' -> 'two to three seconds'
Root cause
en/data/measure/unit.tsv: lbs pound at L45, min minute at L59 (duplicated at L129), sec at L128, hr at L130 — no plural written forms.
No grammar change is needed to support them: MeasureFst already derives the spoken plural from the singular value via measure.py L74 (graph_unit_plural = convert_space(graph_unit @ SINGULAR_TO_PLURAL)), so a plural written key only needs a row mapping to the singular spoken form.
Proposed fix (
Append to unit.tsv, mirroring the existing lbs → pound precedent:
mins minute
hrs hour
secs second
Verified locally against 1.2.0
'5 mins' -> 'five minutes'
'2-3 mins' -> 'two to three minutes'
'2-3 hrs' -> 'two to three hours'
'30-45 secs' -> 'thirty to forty five seconds'
'1 min' -> 'one minute' # singular side unaffected
'2013-2016' -> 'twenty thirteen to twenty sixteen' # no regressions observed
- More generally: allow callers to extend the unit vocabulary at
Normalizerconstruction, e.g.
Normalizer(input_case="cased", lang="en", extra_unit_file="my_units.tsv")
threaded down to MeasureFst, where the user file is unioned with (not substituted for) the shipped one:
graph_unit = pynini.string_file(get_abs_path("data/measure/unit.tsv"))
if extra_unit_file:
graph_unit |= pynini.string_file(extra_unit_file) # append: built-ins kept
Today there is no supported way to add a unit without editing the installed package's data files and regenerating the FAR cache by hand. The existing whitelist= parameter doesn't cover this: in deterministic mode a user-supplied whitelist replaces the built-in whitelist rather than merging (en/taggers/whitelist.py L124-L128), and whitelist entries are plain string maps that don't participate in measure semantics (pluralization, get_range's "2-3 km" → "two to three kilometers"). An append-style unit hook would let users carry domain vocabulary (svc → service, locale-specific abbreviations, …) without forking data files. The FAR cache filename would need to account for the extra file (e.g. a content-hash suffix), since the compiled FST depends on it.
Environment details
- Environment location: bare-metal Linux
- Method of NeMo install:
pip install nemo-text-processing==1.2.0 - Python 3.13.x, pynini 2.1.6.post1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with nemo_text_processing/text_normalization/en/data/measure/unit.tsv and compare its unit mappings with en/taggers/measure.py, especially the pluralization path. Run the reproduction snippet from the issue before and after the focused data change. Done means plural mins, hrs, and secs normalize correctly in single values and ranges without changing singular or year-range behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100