internetarchive / internetarchive/surt

Keep non-UTF-8 encoded URLs (Python 3)

Open
#19 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
18
PR merge metrics
No merged PRs in 30d

Description

Surt with Python 2.x keeps URLs with non-ASCII characters in the percent-encoded path intact:

http://onlinestreet.de/strassen/in-M%FCnchen.html

(letters in hex characters are lowercased)

de,onlinestreet)/strassen/in-m%fcnchen.html

With Python 3.x the latin-1-encoded character is substituted by the replacement character:

de,onlinestreet)/strassen/in-m%ef%bf%bdnchen.html
http://onlinestreet.de/strassen/in-m%ef%bf%bdnchen.html

Since UTF-8 as character encoding was introduced by [RFC3986](https://tools.ietf.org/html/rfc3986) in 2005, there may be still many URLs which use a different encoding.

2d4bde5 adds a test to catch this problem.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Python 3 SURT conversion with the non-UTF-8 URL shown in the issue and compare it with the Python 2 result. Review the test added by commit 2d4bde5, then verify that the percent-encoded path remains intact and its hexadecimal letters are lowercased.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.