saltstack / saltstack/salt

[BUG] mysql returner: MYSQLdb use 'latin-1' encoding by default but tables are utf-8

Open
#56,895 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core needs-testcase Returners severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When using mysql returner, there's a traceback when saving some events because MYSQLdb use 'latin-1' encoding by default but tables are utf-8.

Setup
A simple master minion setup with a database.

Steps to Reproduce the behavior
try to run sys.list_functions, sys.doc, sys.list_runner_functions and sys.runner_doc on the minion.

master_1  | 10:33:38 master.1 | [ERROR   ] Error in function _return:
master_1  | 10:33:38 master.1 | Traceback (most recent call last):
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/salt/master.py", line 1839, in run_func
master_1  | 10:33:38 master.1 |     ret = getattr(self, func)(load)
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/salt/master.py", line 1638, in _return
master_1  | 10:33:38 master.1 |     self.opts, load, event=self.event, mminion=self.mminion)
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/salt/utils/job.py", line 108, in store_job
master_1  | 10:33:38 master.1 |     mminion.returners[savefstr](load['jid'], load)
master_1  | 10:33:38 master.1 |   File "/var/cache/salt/master/extmods/returners/mysql.py", line 362, in save_load
master_1  | 10:33:38 master.1 |     cur.execute(sql, (jid, salt.utils.json.dumps(load)))
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 207, in execute
master_1  | 10:33:38 master.1 |     args = tuple(map(db.literal, args))
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 304, in literal
master_1  | 10:33:38 master.1 |     s = self.escape(o, self.encoders)
master_1  | 10:33:38 master.1 |   File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in unicode_literal
master_1  | 10:33:38 master.1 |     return db.literal(str(u).encode(unicode_literal.charset))
master_1  | 10:33:38 master.1 | UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 73212: ordinal not in range(256)
master_1  | 10:33:38 master.1 | [ERROR   ] Could not store events - returner 'mysql.event_return' raised exception: 'latin-1' codec can't encode character '\u2019' in position 73469: ordinal not in range(256)

Expected behavior
No traceback pls.

Versions Report

Salt Version:
           Salt: 3000.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
         Jinja2: 2.9.4
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.11 
         locale: UTF-8
        machine: x86_64
        release: 5.5.15-200.fc31.x86_64
         system: Linux
        version: debian 9.11 

MYSQLdb should be forced to use 'utf-8' as an encoding.
adding options:

charset="utf8",
use_unicode=True,

to the MySQLdb.connect function do the trick.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the MySQL returner connection setup, then trace the save_load path shown in the traceback. Reproduce the issue with the listed sys commands and verify that events containing non-Latin-1 characters are stored without a traceback when the connection uses the requested UTF-8 options.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.