apache / apache/stormcrawler

Politeness keys do not collapse IDN host spellings (unicode vs punycode)

Open Beginner friendly
#2,145 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

Follow-up from #2122 (review by rzo1).

okhttp punycodes the host when it parses a URL: `IDN.toASCII("ExAmpleé.org")` gives `xn--exampl-gva.org`. `URLUtil.getCanonicalHost` (introduced in #2122 for politeness queues and the robots.txt cache) lowercases and percent-decodes the host but does not punycode it, so the Unicode and punycode spellings of one server still get separate politeness queues and separate robots.txt cache entries:

- `http://ExAmpleé.org/` → host `ExAmpleé.org` → key `examplé.org`
- `http://xn--exampl-gva.org/` → key `xn--exampl-gva.org`

Suggested fix: apply `IDN.toASCII` in `getCanonicalHost` (falling back to the raw host when the label is not a valid IDN), so both spellings collapse the way okhttp collapses them at connect time.

Contributor guide

Open the contributing guide

Research direction

Start at URLUtil.getCanonicalHost and compare the canonical keys produced for the Unicode and punycode host examples in the issue. Check how okhttp handles IDN.toASCII and preserve the raw host for invalid labels. Done means both valid spellings produce one key while invalid hosts still retain their raw value; add or update focused tests for these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.