[BUG] mysql returner: MYSQLdb use 'latin-1' encoding by default but tables are utf-8
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When using mysql returner, there's a traceback when saving some events because MYSQLdb use 'latin-1' encoding by default but tables are utf-8.
Setup
A simple master minion setup with a database.
Steps to Reproduce the behavior
try to run sys.list_functions, sys.doc, sys.list_runner_functions and sys.runner_doc on the minion.
master_1 | 10:33:38 master.1 | [ERROR ] Error in function _return:
master_1 | 10:33:38 master.1 | Traceback (most recent call last):
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/salt/master.py", line 1839, in run_func
master_1 | 10:33:38 master.1 | ret = getattr(self, func)(load)
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/salt/master.py", line 1638, in _return
master_1 | 10:33:38 master.1 | self.opts, load, event=self.event, mminion=self.mminion)
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/salt/utils/job.py", line 108, in store_job
master_1 | 10:33:38 master.1 | mminion.returners[savefstr](load['jid'], load)
master_1 | 10:33:38 master.1 | File "/var/cache/salt/master/extmods/returners/mysql.py", line 362, in save_load
master_1 | 10:33:38 master.1 | cur.execute(sql, (jid, salt.utils.json.dumps(load)))
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 207, in execute
master_1 | 10:33:38 master.1 | args = tuple(map(db.literal, args))
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 304, in literal
master_1 | 10:33:38 master.1 | s = self.escape(o, self.encoders)
master_1 | 10:33:38 master.1 | File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in unicode_literal
master_1 | 10:33:38 master.1 | return db.literal(str(u).encode(unicode_literal.charset))
master_1 | 10:33:38 master.1 | UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 73212: ordinal not in range(256)
master_1 | 10:33:38 master.1 | [ERROR ] Could not store events - returner 'mysql.event_return' raised exception: 'latin-1' codec can't encode character '\u2019' in position 73469: ordinal not in range(256)
Expected behavior
No traceback pls.
Versions Report
Salt Version:
Salt: 3000.1
Dependency Versions:
cffi: Not Installed
cherrypy: 3.5.0
dateutil: 2.5.3
docker-py: Not Installed
gitdb: 2.0.0
gitpython: 2.1.1
Jinja2: 2.9.4
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: 1.3.7
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
smmap: 2.0.1
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.11
locale: UTF-8
machine: x86_64
release: 5.5.15-200.fc31.x86_64
system: Linux
version: debian 9.11
MYSQLdb should be forced to use 'utf-8' as an encoding.
adding options:
charset="utf8",
use_unicode=True,
to the MySQLdb.connect function do the trick.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the MySQL returner connection setup, then trace the save_load path shown in the traceback. Reproduce the issue with the listed sys commands and verify that events containing non-Latin-1 characters are stored without a traceback when the connection uses the requested UTF-8 options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100