UAX_URL_EMAIL tokenizer not compliant to rfc1808 [LUCENE-8044]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I noticed that the uax_url_email tokenizer splits urls in multiple tokens in presence of digits, ".", "-"
I opened a issue on elasticsearch github repo (https://github.com/elastic/elasticsearch/issues/27309) because I noticed this strange behaviour.
Their answer was
> The uax_url_email tokenizer tokenizes URLs and email addresses, but in order to recognize a token as a URL it must include the scheme (usually HTTP:// or https://):
> Additionally, this tokenizer belongs to Lucene\. Could you open this issue at https://lucene\.apache\.org/core/ instead?
URLs are defined by RFC1738 and extended by RFC1808.
In RFC1808 Relative URLs are explained, and this allows scheme-less URLs.
I would expect uax_url_email to tokenize correctly also scheme-less and relative URL.
---
Migrated from [LUCENE-8044](https://issues.apache.org/jira/browse/LUCENE-8044) by Sergio Leoni
Environment:
```
Elasticsearch 5.5.2, Build: b2f0c09/2017-08-14T12:33:14.154Z, JVM: 1.8.0_144
JVM java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
OS Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
```
Contributor guide
Research direction
Start by locating the UAX_URL_EMAIL tokenizer and reviewing its handling of digits, periods, hyphens, scheme-less URLs, and relative URLs. Compare the current behavior with RFC1738 and RFC1808 and the Elasticsearch example linked in the issue. Done means valid scheme-less and relative URLs are tokenized as expected, with regression coverage for the reported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100