[BUG] UnicodeEncodeError when using mysql.file_query in Python3 Minion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 53-54: ordinal not in range(256) if there are Unicode characters in the file to be used by mysql.file_query.
The minion function caused an exception: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1936, in _thread_return
function_name, function_args, executors, opts, data
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1894, in _execute_job_function
return_data = self.executors[fname](opts, data, func, args, kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 882, in file_query
query_result = query(database, query_string, **connection_args)
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 784, in query
affected = _execute(cur, query)
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 668, in _execute
return cur.execute(qry)
File "/usr/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 243, in execute
query = query.encode(db.encoding, 'surrogateescape')
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 53-54: ordinal not in range(256)
Setup
$ locale
LANG=C
LC_CTYPE="ko_KR.UTF-8"
LC_NUMERIC="ko_KR.UTF-8"
LC_TIME="ko_KR.UTF-8"
LC_COLLATE="ko_KR.UTF-8"
LC_MONETARY="ko_KR.UTF-8"
LC_MESSAGES="ko_KR.UTF-8"
LC_PAPER="ko_KR.UTF-8"
LC_NAME="ko_KR.UTF-8"
LC_ADDRESS="ko_KR.UTF-8"
LC_TELEPHONE="ko_KR.UTF-8"
LC_MEASUREMENT="ko_KR.UTF-8"
LC_IDENTIFICATION="ko_KR.UTF-8"
LC_ALL=ko_KR.UTF-8
$ echo $PYTHONIOENCODING
utf-8:surrogateescape
Steps to Reproduce the behavior
salt 'target' mysql.file_query test_db file_name=salt://push2.sql
push2.sql
CREATE TABLE push (
id BIGINT(20) NOT NULL COMMENT '번호',
push_id BIGINT(20) NULL COMMENT '푸시ID'
) COMMENT '푸시 전송 내역';
ALTER TABLE push ADD CONSTRAINT ipk_push PRIMARY KEY (id ASC);
Expected behavior
Screenshots
Versions Report
Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: 1.3.12
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: utf-8
machine: x86_64
release: 3.10.0-1160.76.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/modules/mysql.py, especially file_query, query, and _execute, then reproduce the failure using the provided mysql.file_query command and SQL containing Unicode characters. Compare the query path with the reported MySQLdb cursor failure. Done means the same SQL file can be processed without UnicodeEncodeError and the relevant behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100