Remove Telugu normalization of vu వు to ma మ from IndicNormalizer
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Telugu vu వు and ma మ are visually similar—akin to English "rn" and "m"—but they should not be conflated. Names like వెంకటరామ (Venkatarama) and వెంకటరావు (Venkatarao) and words like [మండే](https://te.wiktionary.org/wiki/%E0%B0%AE%E0%B0%82%E0%B0%A6%E0%B0%BF) and [వుండే](https://te.wiktionary.org/wiki/%E0%B0%B5%E0%B1%81%E0%B0%82%E0%B0%A6%E0%B0%BF) (links to Telugu Wiktionary) are distinct.
It's like conflating "rn" and "m" to merge _burn/bum_ and _corn/com._ It could happen when reading quickly or with poor handwriting, but it is not something that should happen for search indexing.
I notice that some of the Telugu elements of IndicNormalizer are in TeluguNormalizer, but this mapping is not—which is good!
(Sorry for the botched pull request. Obviously this change would also affect some tests, which need to be updated or re-evaluated.)
### Version and environment details
My version:
"distribution" : "opensearch",
"number" : "1.3.20",
"lucene_version : "8.10.1"
Running on x86_64 GNU/Linux in Docker 4.15.0 on MacOS 13.6.3.
Contributor guide
Research direction
Start by locating the Telugu mapping in IndicNormalizer and compare it with TeluguNormalizer, which the issue says does not contain this mapping. Find the affected normalization tests and re-evaluate them so Telugu vu వు and ma మ remain distinct; done means the unwanted normalization is removed and the tests reflect that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100