UnicodeDecodeError when reading ANSI files
- Dominant language
- Python
- Stars
- 172
- Forks
- 41
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 1
Description
If circup tries to find the version information in a `.py` file that has ANSI encoded characters like `‘’`, it throws [here](https://github.com/adafruit/circup/blob/main/circup/shared.py#L120C1-L121C41):
```
Found device at D:\, running CircuitPython 9.1.0-beta.3.
Searching for dependencies for: ['Adafruit_CircuitPython_esp32spi']
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "C:\dev\adafruit\envs\circup\Scripts\circup.exe\__main__.py", line 7, in
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\click\decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\commands.py", line 347, in install
device_modules = ctx.obj["backend"].get_device_versions()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\backends.py", line 73, in get_device_versions
return self.get_modules(os.path.join(self.device_location, self.LIB_DIR_PATH))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\backends.py", line 63, in get_modules
return self._get_modules(device_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\backends.py", line 806, in _get_modules
return _get_modules_file(device_lib_path, self.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\shared.py", line 87, in _get_modules_file
metadata = extract_metadata(source, logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\dev\adafruit\envs\circup\Lib\site-packages\circup\shared.py", line 121, in extract_metadata
content = source_file.read()
^^^^^^^^^^^^^^^^^^
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 5007: invalid start byte
```
This could be changed from:
```py
with open(path, "r", encoding="utf-8") as source_file:
content = source_file.read()
```
To:
```py
try:
with open(path, "r", encoding="utf-8") as source_file:
content = source_file.read()
except UnicodeDecodeError:
with open("a.txt", "r") as source_file:
content = source_file.read()
```
Contributor guide
Research direction
Start in circup/shared.py at extract_metadata and reproduce the failure with a Python file containing ANSI-encoded characters. Check how version metadata is extracted and verify that reading such a file no longer raises UnicodeDecodeError while the existing metadata lookup still completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100