internetarchive / internetarchive/surt

Different URLs that contains “www” in the host name are incorrectly canonicalized.

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
18
PR merge metrics
No merged PRs in 30d

Description

The following URLs are incorrectly canonicalized with SURT as "com)/".
```
SURT = "com)/"
1. https://www1355544.com/
2. https://www3288.com/
3. https://www504778.com/
4. https://www556798.com/
5. https://www57912.com/
```

Problem is that 'www\d*\.' is removed from the URL, which leaves "com)/" as the SURT.
https://github.com/internetarchive/surt/blob/6934c321b3e2f66af9c001d882475949f00570c5/surt/IAURLCanonicalizer.py#L127

the other instance where the URLs were correctly canonicalized by removing 'www\d*\.'
```
SURT = "jp,co,daily)/"
1. https://daily.co.jp/
2. https://www4.daily.co.jp/
3. https://www8.daily.co.jp/
4. https://www9.daily.co.jp/
```
We proposed to update the regex where it should only remove the 'www\d*\.' if at least two dots are present in the hostname.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in surt/IAURLCanonicalizer.py around line 127, where the www-number hostname prefix is removed. Check the regex against the listed single-label and multi-label hostnames, then verify that only the intended prefixes are removed and the resulting SURT values are correct for all examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.