pygments / pygments/pygments

Incomplete remediation of CVE-2021-27291: ScalaLexer ReDoS still present in 2.20.0

Open
#3,247 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.2k
Forks
885
PR merge metrics
No merged PRs in 30d

Description

Incomplete remediation of CVE-2021-27291 / CVE-2021-20272: ScalaLexer ReDoS still present in Pygments 2.20.0

Summary

A catastrophic-regex (ReDoS) hang remains in the ScalaLexer (jvm.py) of the current Pygments release (2.20.0). This is the same string-literal ReDoS class that was fixed across ~25 lexers in 2.7.4 for CVE-2021-27291 / CVE-2021-20272, but the scala string-escape lexing path was missed (incomplete remediation — a surviving unpatched sibling).

Minimal repro

from pygments.lexers import get_lexer_by_name
from pygments import highlight
from pygments.formatters import NullFormatter
lex = get_lexer_by_name("scala")
highlight(b'"' + b"\\\\" * 48, lex, NullFormatter())   # unterminated double-quoted string of 48 backslashes

Timing (single core, Python 3.12)

input time
b'"' + b'\\'*16 0.001 s
b'"' + b'\\'*32 0.38 s
b'"' + b'\\'*48 > 6 s (hangs)
terminated control b'"' + b'\\'*128 + b'"' 0.000 s
non-backslash control b'"' + b'a'*128 0.000 s

Backslash-dependent, super-exponential with N, requires the unterminated (forced-backtrack) form. A ~50-byte input hangs the highlighter.

Class / causality

The hang is catastrophic regex backtracking in the scala string backslash-escape lexing on an unterminated quote. The 2.7.4 CVE fix replaced the ambiguous (\\\\|\\"|[^"])* family with the linear-safe (\\\\|\\[^\\\\]|[^"\\\\])* form in the reported lexers; the scala string/escape path (jvm.py ScalaLexer) does not appear to use the hardened form for this sub-state. Maintainer triage is requested to locate the exact regex in the ScalaLexer string states (I could not isolate a single standalone-compiling token — it manifests only through the lexer's incremental per-position matching).

Proposed fix (2.7.4-style)

Harden the scala string backslash-escape regex to the non-overlapping form, e.g. replace
\\\.? / ambiguous (\\\\|\\"|[^"])* patterns with the atomic-safe
(\\\\|\\[^\\\\]|[^"\\\\])* or add (?>...) / possessive quantifiers so backslashes cannot be re-partitioned.

Regression test (timing)

Assert that lexing b'"' + b'\\\\'*512 (unterminated) completes in well under 1s in CI for the scala lexer.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with jvm.py and the ScalaLexer string states, using the get_lexer_by_name("scala") and highlight() reproduction from the issue. Trace the unterminated backslash-heavy input, then add a regression test showing that 512 backslashes complete promptly. Done means the ScalaLexer no longer exhibits catastrophic backtracking while normal string lexing remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, security, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.