mapbox / mapbox/osm-bright

imposm / osm-bright fails with UnicodeEncodeError

Open
#64 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
CartoCSS
Stars
703
Forks
222
PR merge metrics
No merged PRs in 30d

Description

Using imposm fails with an UnicodeEncodeError when processing relations.

I processed an OSM file from Geofabrik (alps.osm.pbf). At the end it failed with the following error repeated 4 times, and the process hanged.

Last message was: relations: 10k/79k. TileMill complains about missing relations.

$ imposm -U postgres -d osm -m mapbox-osm-bright-c9bc5ee/imposm-mapping.py --read --write --optimize --deploy-production-tables alps_20130208.osm.pbf 
loading mapbox-osm-bright-c9bc5ee/imposm-mapping.py as mapping
password for postgres at localhost:
[11:38:55] ## reading alps_20130208.osm.pbf
[11:38:55] coords: 78954k nodes: 1579k ways: 11279k relations: 78k (estimated)
[11:51:19] coords: 103636k nodes: 365k ways: 12045k relations: 73k
[11:51:21] reading took 12m 25s
[11:51:21] ## dropping/creating tables
[11:51:25] ## writing data
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/imposm/dbimporter.py", line 87, in db_importer
    db.insert(mapping, insert_data)
  File "/usr/local/lib/python2.7/dist-packages/imposm/db/postgis.py", line 104, in insert
    self.cur.executemany(insert_stmt, insert_data)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 9: ordinal not in range(128)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback locations in imposm/dbimporter.py and imposm/db/postgis.py, then reproduce the failure with the shown imposm command and alps_20130208.osm.pbf input. Trace the relation-writing path and verify that processing completes without the UnicodeEncodeError or hang, with relations available to TileMill.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.