saltstack / saltstack/salt

[BUG] UnicodeEncodeError when using mysql.file_query in Python3 Minion

Open
#63,407 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Execution-Module needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 53-54: ordinal not in range(256) if there are Unicode characters in the file to be used by mysql.file_query.

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1936, in _thread_return
        function_name, function_args, executors, opts, data
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1894, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 882, in file_query
        query_result = query(database, query_string, **connection_args)
      File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 784, in query
        affected = _execute(cur, query)
      File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 668, in _execute
        return cur.execute(qry)
      File "/usr/lib64/python3.6/site-packages/MySQLdb/cursors.py", line 243, in execute
        query = query.encode(db.encoding, 'surrogateescape')
    UnicodeEncodeError: 'latin-1' codec can't encode characters in position 53-54: ordinal not in range(256)

Setup

$ locale
LANG=C
LC_CTYPE="ko_KR.UTF-8"
LC_NUMERIC="ko_KR.UTF-8"
LC_TIME="ko_KR.UTF-8"
LC_COLLATE="ko_KR.UTF-8"
LC_MONETARY="ko_KR.UTF-8"
LC_MESSAGES="ko_KR.UTF-8"
LC_PAPER="ko_KR.UTF-8"
LC_NAME="ko_KR.UTF-8"
LC_ADDRESS="ko_KR.UTF-8"
LC_TELEPHONE="ko_KR.UTF-8"
LC_MEASUREMENT="ko_KR.UTF-8"
LC_IDENTIFICATION="ko_KR.UTF-8"
LC_ALL=ko_KR.UTF-8

$ echo $PYTHONIOENCODING
utf-8:surrogateescape

Steps to Reproduce the behavior

salt 'target' mysql.file_query test_db file_name=salt://push2.sql

push2.sql
CREATE TABLE push (
        id BIGINT(20) NOT NULL COMMENT '번호',
        push_id BIGINT(20) NULL COMMENT '푸시ID'
) COMMENT '푸시 전송 내역';

ALTER TABLE push ADD CONSTRAINT ipk_push PRIMARY KEY (id ASC);

Expected behavior

Screenshots

Versions Report

Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: 1.3.12
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 18.0.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.76.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/modules/mysql.py, especially file_query, query, and _execute, then reproduce the failure using the provided mysql.file_query command and SQL containing Unicode characters. Compare the query path with the reported MySQLdb cursor failure. Done means the same SQL file can be processed without UnicodeEncodeError and the relevant behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.