Politeness keys do not collapse IDN host spellings (unicode vs punycode)
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
Follow-up from #2122 (review by rzo1).
okhttp punycodes the host when it parses a URL: `IDN.toASCII("ExAmpleé.org")` gives `xn--exampl-gva.org`. `URLUtil.getCanonicalHost` (introduced in #2122 for politeness queues and the robots.txt cache) lowercases and percent-decodes the host but does not punycode it, so the Unicode and punycode spellings of one server still get separate politeness queues and separate robots.txt cache entries:
- `http://ExAmpleé.org/` → host `ExAmpleé.org` → key `examplé.org`
- `http://xn--exampl-gva.org/` → key `xn--exampl-gva.org`
Suggested fix: apply `IDN.toASCII` in `getCanonicalHost` (falling back to the raw host when the label is not a valid IDN), so both spellings collapse the way okhttp collapses them at connect time.
Contributor guide
Research direction
Start at URLUtil.getCanonicalHost and compare the canonical keys produced for the Unicode and punycode host examples in the issue. Check how okhttp handles IDN.toASCII and preserve the raw host for invalid labels. Done means both valid spellings produce one key while invalid hosts still retain their raw value; add or update focused tests for these cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100