Bug: UnicodeEncodeError in create_provider_plugin.py on Windows (cp1252 console)
- Dominant language
- Python
- Stars
- 38.6k
- Forks
- 2.7k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 4
Description
## Describe the overall issue and situation
On Windows, `scripts/create_provider_plugin.py` and the `test_plugin.py` it generates print Unicode checkmarks (`✓` / `✅`). Windows consoles default stdout/stderr to a legacy code page (typically cp1252), so both scripts crash with `UnicodeEncodeError` before the plugin can be generated or verified.
This is the same root-cause class as #108, #158, and #166 (Windows cp1252 vs Unicode), which were fixed for file I/O / HTML export in `io.py`. The generator script's console output was not covered.
Related to #158 (6 👍; already-fixed HTML/file I/O case). This issue is the remaining console-output path.
## Expected behavior
`python scripts/create_provider_plugin.py ...` and the generated `test_plugin.py` print status lines successfully on a default Windows console.
## Actual behavior
```
UnicodeEncodeError: 'charmap' codec can't encode character '\u2713' in position 0: character maps to
```
`pytest tests/create_provider_plugin_test.py` also fails on Windows because `subprocess.run(..., text=True)` decodes the generated script's UTF-8 output with the locale encoding, producing mojibake instead of the expected checkmarks.
## Steps to reproduce the issue
1. On Windows, open a default `cmd.exe` or PowerShell session (code page 1252 / cp1252).
2. From a clone: `python scripts/create_provider_plugin.py --help` or run the generator as documented.
3. Observe `UnicodeEncodeError` when the script prints a `✓` / `✅` status line.
4. Alternatively: `pytest tests/create_provider_plugin_test.py` — 15 failures on Windows before a UTF-8 stdout/stderr reconfigure.
## Any additional content
- OS: Windows
- Python: 3.14 (also reproduces on other Windows CPython versions with a non-UTF-8 console)
- Fix proposed in https://github.com/google/langextract/pull/514 (reconfigure stdout/stderr to UTF-8; decode subprocess output as UTF-8)
- Environment: default Windows console, not Windows Terminal UTF-8 mode
Contributor guide
Research direction
Start with scripts/create_provider_plugin.py and the generated test_plugin.py, then inspect tests/create_provider_plugin_test.py and its subprocess handling. Run the focused pytest suite on Windows with a cp1252 console; done means both scripts emit status lines without UnicodeEncodeError and the tests receive the expected checkmarks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100