collective / collective/transmogrify.wordpress
URLs with extended characters lead to errors and inconsistencies
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I'm working to import a site which contains some extended characters on URLs like this:
http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretário-de-Negócios-Jurídicos-Sérgio-Médici-09jan17-TeTê-Viviani-1024x769.jpg
this leads to an `UnicodeDecodeError` on the `FetchAttachment` section when the resource is missing, as the `url` variable was supposed to be a string:
```python
if r.status_code != 200: # log error and skip item
fetch_errors.append(url)
msg = u'Error {0} when fetching {1}'.format(r.status_code, url)
logger.warn(msg)
yield item
continue
```
also, the ids of the resulting object are inconsistent with the original names:
```python
>> url.decode('utf-8').encode('ascii', 'ignore')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretrio-de-Negcios-Jurdicos-Srgio-Mdici-09jan17-TeT-Viviani-1024x769.jpg'
>>> url.decode('utf-8').encode('ascii', 'replace')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secret?rio-de-Neg?cios-Jur?dicos-S?rgio-M?dici-09jan17-TeT?-Viviani-1024x769.jpg'
>>> url.decode('utf-8').encode('ascii', 'xmlcharrefreplace')
'http://www.araraquara.sp.gov.br/wp-content/uploads/2017/01/foto08-Prefeito-Edinho-Silva-empossa-secretário-de-Negócios-Jurídicos-Sérgio-Médici-09jan17-TeTê-Viviani-1024x769.jpg'
```
it would be best to use the Plone id normalizer.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.