internetarchive / internetarchive/surt
Keep non-UTF-8 encoded URLs (Python 3)
- Dominant language
- Python
- Stars
- 47
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Surt with Python 2.x keeps URLs with non-ASCII characters in the percent-encoded path intact:
http://onlinestreet.de/strassen/in-M%FCnchen.html
(letters in hex characters are lowercased)
de,onlinestreet)/strassen/in-m%fcnchen.html
With Python 3.x the latin-1-encoded character is substituted by the replacement character:
de,onlinestreet)/strassen/in-m%ef%bf%bdnchen.html
http://onlinestreet.de/strassen/in-m%ef%bf%bdnchen.html
Since UTF-8 as character encoding was introduced by [RFC3986](https://tools.ietf.org/html/rfc3986) in 2005, there may be still many URLs which use a different encoding.
2d4bde5 adds a test to catch this problem.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Python 3 SURT conversion with the non-UTF-8 URL shown in the issue and compare it with the Python 2 result. Review the test added by commit 2d4bde5, then verify that the percent-encoded path remains intact and its hexadecimal letters are lowercased.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100