openedx / openedx/openedx-platform

Invalid characters in unit title break the course structure

Open
#31,301 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.2k
Forks
4.4k
Avg merge
6d 18h
Merged PRs (30d)
42

Description

Description

When copy-pasting text from some sources into the section, subsection or unit titles, some invisible characters may be inadvertently inserted. When the backend tries to insert the record, the database will reject the operation resulting in a broken course structure.

How to reproduce

In a python console, create a string with a '\u200b' character at the end. Print the result (the special character will not be visible, but will still be there). Select the printed text and copy to the clipboard. Then paste into a section title. Watch the CMS logs.

>>> t='Test\u200b'
>>> t
'Test\u200b'
>>> print(t)
Test​
image

CMS logs:

[2022-11-16 15:16:20,036: ERROR/ForkPoolWorker-43] cms.djangoapps.contentstore.tasks.update_outline_from_modulestore_task[9288311e-d729-4133-9eca-19378df3aff5]: Could not create course outline for course course-v1:TEST+TEST+2022
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
openedx.core.djangoapps.content.learning_sequences.models.LearningSequence.DoesNotExist: LearningSequence matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\xE2\\x80\\x8B' for column 'title' at row 1")
Expected result

Invalid characters should be removed from the titles before inserting into the backend.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the logged cms.djangoapps.contentstore.tasks.update_outline_from_modulestore_task entry point and trace how section, subsection, or unit titles reach the database. Reproduce the issue with a title containing \u200b, then add coverage showing invalid characters are removed and the course structure is created successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.