GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

UnicodeEncodeError: 'charmap' codec can't encode character exception on Windows

Open Beginner friendly
#1,773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

## Example error
```
File "c:\Users\x\dev\DVT\Lib\site-packages\data_validation\util.py", line 32, in timed_call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "c:\Users\x\dev\DVT\Lib\site-packages\data_validation\result_handlers\text.py", line 75, in print_formatted_
print(self._get_formatted(result_df))
File "c:\Users\x\dev\DVT\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\x81' in position 351: character maps to
```

## Suggested workaround from Windows Command Prompt (CMD):
```
set PYTHONIOENCODING=utf-8
data-validation validate row -sc=c1 -tc=c2 --tbls ... > "%output_dir%\dbo_tblbank.csv"
```

## Potential code change
To prevent crashes on Windows when printing non-ASCII / UTF-8 characters to redirected standard output, `text.py` can safely reconfigure standard output to UTF-8 before calling `print` :
```
import sys

# Ensure stdout uses UTF-8 encoding so redirected files don't fail on Windows cp1252
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8")
```

We should either document the workaround or implement the potential solution.

Contributor guide

Open the contributing guide

Research direction

Start in text.py around print_formatted_ and reproduce the Windows failure using the documented data-validation validate command with redirected output. Check whether the command still fails with non-ASCII text, then either document the PYTHONIOENCODING workaround or implement and verify the selected stdout-handling change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.