apache / apache/lucene

UAX_URL_EMAIL tokenizer not compliant to rfc1808 [LUCENE-8044]

Open
#9,092 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.6 legacy-jira-priority:Minor module:core/other type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I noticed that the uax_url_email tokenizer splits urls in multiple tokens in presence of digits, ".", "-"

I opened a issue on elasticsearch github repo (https://github.com/elastic/elasticsearch/issues/27309) because I noticed this strange behaviour.

Their answer was
> The uax_url_email tokenizer tokenizes URLs and email addresses, but in order to recognize a token as a URL it must include the scheme (usually HTTP:// or https://):
> Additionally, this tokenizer belongs to Lucene\. Could you open this issue at https://lucene\.apache\.org/core/ instead?

URLs are defined by RFC1738 and extended by RFC1808.
In RFC1808 Relative URLs are explained, and this allows scheme-less URLs.
I would expect uax_url_email to tokenize correctly also scheme-less and relative URL.

---
Migrated from [LUCENE-8044](https://issues.apache.org/jira/browse/LUCENE-8044) by Sergio Leoni
Environment:
```
Elasticsearch 5.5.2, Build: b2f0c09/2017-08-14T12:33:14.154Z, JVM: 1.8.0_144

JVM java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS Linux 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the UAX_URL_EMAIL tokenizer and reviewing its handling of digits, periods, hyphens, scheme-less URLs, and relative URLs. Compare the current behavior with RFC1738 and RFC1808 and the Elasticsearch example linked in the issue. Done means valid scheme-less and relative URLs are tokenized as expected, with regression coverage for the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.