International Domain Names (IDNs) are not presented/handled consistently across the plugin
- Dominant language
- JavaScript
- Stars
- 1.4k
- Forks
- 383
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 80
Description
## Bug Description
Site Kit should allow users to set up Site Kit when they have an [IDN](https://en.wikipedia.org/wiki/Internationalized_domain_name), eg. a domain with Unicode/non-ASCII characters.
We support this in theory, but it seems like our implementation (along—potentially—with some of WordPress's handling of IDNs) is inconsistent, leading to minor bugs like failing to redirect properly (#5868) and possibly failing to create an Analytics account via Site Kit.
IDNs can be presented in two different ways: either in their "original"/"raw" format, eg. `türkish.com`, or in Punycode eg. `xn--trkish-3ya.com`. While WordPress, the Site Kit service/proxy, and parts of our plugin can handle either, it seems we aren't entirely consistent about how we present them to users or how we submit the domain name/site URL to Google Services like Analytics.
For instance, here's a site with the `siteurl` `türkish.com`, connecting an Analytics account. Note the domain detected/presented is Punycode and that this fails to set up an Analytics account:
https://user-images.githubusercontent.com/90871/213430033-5c8adabb-b044-471d-9e4b-9c05f4694ea9.mp4
Here's the same site after encountering that failed set up, when the domain is manually substituted for the Unicode one:
https://user-images.githubusercontent.com/90871/213430312-24acd253-3f7e-403a-b8a1-f33a1a86df19.mp4
If we set the `siteurl` to a Punycode string instead of a Unicode one, note that the Punycode Analytics account creation works just fine:
https://user-images.githubusercontent.com/90871/213432130-dbec3705-9f29-4cd4-b421-ee649679a286.mp4
---------------
It's worth noting that because we treat the Unicode and Punycode domains as different, there's also this odd error users can encounter when changing between the Unicode and non-Unicode `siteurl`:

Arguably that isn't really true, and we should probably consistently treat all IDNs as either Unicode or Punycode in our codebase and always present them as Unicode in the UI.
---------------
_Do not alter or remove anything below. The following sections will be managed by moderators only._
## Acceptance criteria
* The existing `URL` class should be extended to allow creation of `URL` instances which hold a WordPress/site URL (or any other URL) created from a string of either ASCII, punycode, or Unicode characters.
* This class should accept a string as (part of) its constructor, and convert the string internally to an ASCII domain. Eg. If the string is already plain ASCII, like `google.com`, the string would be stored as-is. If the string is Unicode (`türkish.com`), it should be converted to a punycode string (`xn--trkish-3ya.com`).
* The class should offer two methods to output the URL: "user-facing" and "internal". The user-facing one should _always_ be the "pretty", Unicode version, regardless of whether the input string was punycode or Unicode. The "internal" version should _always_ be the punycode version.
* Going forward, Site Kit should handle all URL strings by converting them into a `URL` instance and using the correct output method based on the requirements. In PHP code, most usage is internal and not part of user-facing UI, so this will almost always be the `$url->internal()` format.
* In JS, we should use the punycode URLs whenever dealing with non-user-facing URLs. Because some Google APIs could return Unicode domains/URLs in their responses, we may also need to convert URLs received from APIs with a JS `normalizeURL( url )` type utility that normalises the URL into a punycode version.
* In JS, we should always convert URLs from punycode to Unicode format when displaying them in UIs, like showing the connected Search Console property URL:

This change of "always using punycode internally" should not pose issues for users who "registered Site Kit" when the internal URL was Unicode and not punycode, because the Site ID is the relevant link between the plugin and Site Kit service.
However, comparing the "internal", punycode domain with existing Analytics properties (or other service's properties) may cause issues where an existing Analytics property exists but for a Unicode version of our "internal" punycode domain.
* This means when comparing existing property URLs with our WordPress URL, we should compare both punycode and Unicode versions to find a match.
## Implementation Brief
*
### Test Coverage
*
## QA Brief
*
## Changelog entry
*
Contributor guide
Research direction
Start by locating the existing URL class and the PHP and JavaScript URL-handling entry points mentioned in the acceptance criteria. Trace Analytics account creation and Search Console property URL comparisons, then verify that ASCII, Unicode, and Punycode inputs are normalized consistently for internal use and user-facing display.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, php, wordpress
- Domain
- backend, frontend, internationalization
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100