internetarchive / internetarchive/surt
Different URLs that contains “www” in the host name are incorrectly canonicalized.
- Dominant language
- Python
- Stars
- 47
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
The following URLs are incorrectly canonicalized with SURT as "com)/".
```
SURT = "com)/"
1. https://www1355544.com/
2. https://www3288.com/
3. https://www504778.com/
4. https://www556798.com/
5. https://www57912.com/
```
Problem is that 'www\d*\.' is removed from the URL, which leaves "com)/" as the SURT.
https://github.com/internetarchive/surt/blob/6934c321b3e2f66af9c001d882475949f00570c5/surt/IAURLCanonicalizer.py#L127
the other instance where the URLs were correctly canonicalized by removing 'www\d*\.'
```
SURT = "jp,co,daily)/"
1. https://daily.co.jp/
2. https://www4.daily.co.jp/
3. https://www8.daily.co.jp/
4. https://www9.daily.co.jp/
```
We proposed to update the regex where it should only remove the 'www\d*\.' if at least two dots are present in the hostname.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in surt/IAURLCanonicalizer.py around line 127, where the www-number hostname prefix is removed. Check the regex against the listed single-label and multi-label hostnames, then verify that only the intended prefixes are removed and the resulting SURT values are correct for all examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100