collective / collective/transmogrify.wordpress

URLs with extended characters lead to errors and inconsistencies

Open
#10 0 comments 0 reactions 1 assignee Claimed by @rodfersou View on GitHub
bug
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I'm working to import a site which contains some extended characters on URLs like this:

http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretário-de-Negócios-Jurídicos-Sérgio-Médici-09jan17-TeTê-Viviani-1024x769.jpg

this leads to an `UnicodeDecodeError` on the `FetchAttachment` section when the resource is missing, as the `url` variable was supposed to be a string:

```python
if r.status_code != 200: # log error and skip item
fetch_errors.append(url)
msg = u'Error {0} when fetching {1}'.format(r.status_code, url)
logger.warn(msg)
yield item
continue
```

also, the ids of the resulting object are inconsistent with the original names:

```python
>> url.decode('utf-8').encode('ascii', 'ignore')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretrio-de-Negcios-Jurdicos-Srgio-Mdici-09jan17-TeT-Viviani-1024x769.jpg'
>>> url.decode('utf-8').encode('ascii', 'replace')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secret?rio-de-Neg?cios-Jur?dicos-S?rgio-M?dici-09jan17-TeT?-Viviani-1024x769.jpg'
>>> url.decode('utf-8').encode('ascii', 'xmlcharrefreplace')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretário-de-Negócios-Jurídicos-Sérgio-Médici-09jan17-TeTê-Viviani-1024x769.jpg'
```

it would be best to use the Plone id normalizer.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.