python / python/cpython

Escaping the `repl` argument to `re.sub()`, `re.subn()`

Open
#128,138 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs topic-regex
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Documentation

It's not immediately obvious how to escape the repl (replacement for matches) argument to re.sub() and re.subn() if repl is chosen by a potentially hostile actor. Obviously, re.escape() isn't the answer, as that escapes far too much.

The right answer seems to be escaped_repl = raw_repl.replace(bslash, bslash*2) where bslash = '\\'. It might be worth adding this to the documentation.

Here's the code I used to empirically validate the "right answer" given above (checked on Python 3.8 & 3.12):

from __future__ import annotations
import re, sys

def escape_re_sub_repl(repl: str) -> str:

    return repl.replace('\\', '\\\\')

def test_escape_re_sub_repl() -> None:

    backslash = '\\'
    assert len(backslash) == 1

    base_regex = 'TARGET'
    assert base_regex == re.escape(base_regex)
    base_prefix = 'BEFORE:'
    base_suffix = ':AFTER'
    base_input = f'{base_prefix}{base_regex}{base_suffix}'

    base_chars = tuple(chr(p) for p in range(sys.maxunicode + 1))
    escaped_chars = tuple(f'{backslash}{c}' for c in base_chars)
    test_cases = base_chars + escaped_chars
    assert {len(f) for f in test_cases} == {1, 2}

    for raw in test_cases:
        repl = escape_re_sub_repl(raw)
        got, change_count = re.subn(base_regex, repl, base_input)
        assert change_count == 1
        assert got == f'{base_prefix}{raw}{base_suffix}'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the documentation for re.sub() and re.subn() and its replacement-string rules. Check the existing guidance around re.escape(), then document the backslash-doubling approach for hostile replacement text and validate it against the supplied examples; done means the safe escaping guidance is clear in the relevant documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.