google / google/langextract

Bug: UnicodeEncodeError in create_provider_plugin.py on Windows (cp1252 console)

Open
#518 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38.6k
Forks
2.7k
Avg merge
5d 18h
Merged PRs (30d)
4

Description

## Describe the overall issue and situation

On Windows, `scripts/create_provider_plugin.py` and the `test_plugin.py` it generates print Unicode checkmarks (`✓` / `✅`). Windows consoles default stdout/stderr to a legacy code page (typically cp1252), so both scripts crash with `UnicodeEncodeError` before the plugin can be generated or verified.

This is the same root-cause class as #108, #158, and #166 (Windows cp1252 vs Unicode), which were fixed for file I/O / HTML export in `io.py`. The generator script's console output was not covered.

Related to #158 (6 👍; already-fixed HTML/file I/O case). This issue is the remaining console-output path.

## Expected behavior

`python scripts/create_provider_plugin.py ...` and the generated `test_plugin.py` print status lines successfully on a default Windows console.

## Actual behavior

```
UnicodeEncodeError: 'charmap' codec can't encode character '\u2713' in position 0: character maps to
```

`pytest tests/create_provider_plugin_test.py` also fails on Windows because `subprocess.run(..., text=True)` decodes the generated script's UTF-8 output with the locale encoding, producing mojibake instead of the expected checkmarks.

## Steps to reproduce the issue

1. On Windows, open a default `cmd.exe` or PowerShell session (code page 1252 / cp1252).
2. From a clone: `python scripts/create_provider_plugin.py --help` or run the generator as documented.
3. Observe `UnicodeEncodeError` when the script prints a `✓` / `✅` status line.
4. Alternatively: `pytest tests/create_provider_plugin_test.py` — 15 failures on Windows before a UTF-8 stdout/stderr reconfigure.

## Any additional content

- OS: Windows
- Python: 3.14 (also reproduces on other Windows CPython versions with a non-UTF-8 console)
- Fix proposed in https://github.com/google/langextract/pull/514 (reconfigure stdout/stderr to UTF-8; decode subprocess output as UTF-8)
- Environment: default Windows console, not Windows Terminal UTF-8 mode

Contributor guide

Open the contributing guide

Research direction

Start with scripts/create_provider_plugin.py and the generated test_plugin.py, then inspect tests/create_provider_plugin_test.py and its subprocess handling. Run the focused pytest suite on Windows with a cp1252 console; done means both scripts emit status lines without UnicodeEncodeError and the tests receive the expected checkmarks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, operating-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.