mapbox / mapbox/tilesets-cli

Having trouble reading UTF-8 encoded GeoJSON on Windows

Open
#34 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug enhancement
Dominant language
Python
Stars
141
Forks
27
PR merge metrics
No merged PRs in 30d

Description

I can't figure out what I'm doing wrong. I have a LF delimited geojson I'm trying to upload as a source. The GeoJson contains some non-ANSI characters and so is encoded in UTF-8.

The command I am using looks something like this:
tilesets add-source {account} source-name source-file.geojson

I can't figure out how to get the JSON to parse using UTF-8, it seems like it always tries to use cp1252.py.

So I'm getting this error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5413: character maps to undefined

Full stack trace:

    load_entry_point('tilesets-cli==0.3.2.dev0', 'console_scripts', 'tilesets')()
  File "c:\tools\python\python37\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\tools\python\python37\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\tools\python\python37\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\tools\python\python37\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\tools\python\python37\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\tools\python\python37\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\tools\python\python37\lib\site-packages\tilesets\scripts\cli.py", line 320, in add_source
    for feature in features:
  File "c:\tools\python\python37\lib\site-packages\cligj\features.py", line 30, in normalize_feature_inputs
    for feature in iter_features(iter(src)):
  File "c:\tools\python\python37\lib\site-packages\cligj\features.py", line 121, in iter_features
    text = "".join(chain([first_line], geojsonfile))
  File "c:\tools\python\python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5413: character maps to <undefined>```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at tilesets/scripts/cli.py line 320, where add_source iterates features, then trace the cligj normalize_feature_inputs path shown in the stack trace. Reproduce the Windows UTF-8 GeoJSON failure and determine the intended input-decoding behavior. Done means the command parses the supplied UTF-8 GeoJSON without the cp1252 UnicodeDecodeError, with coverage for the failing case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.