1.7.8 crashes with a tokenize ValueError on a multi-line f-string inside parentheses

docformatter 1.7.8 aborts with a ValueError out of tokenize.untokenize when a file contains a multi-line f-string inside a parenthesized expression. 1.7.7 formats the same file cleanly.

Reproducer

def build(x):
    return (
        f"""a
{x}""",
    )

$ docformatter --in-place t.py
  File ".../docformatter/format.py", line 876, in _do_format_code
    _code = tokenize.untokenize(self.new_tokens)
  File "/usr/lib/python3.11/tokenize.py", line 216, in untokenize
    self.add_whitespace(start)
  File "/usr/lib/python3.11/tokenize.py", line 177, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (2,8) precedes previous end (3,4)
$ echo $?
1

No config file and no flags are needed. Two things narrow it:

• Remove the f prefix and the run exits 3 (would reformat) with no error, so the f-string prefix is the trigger.
• Assign the same f-string to a name instead of putting it in a tuple and it does not crash, so it has to sit inside parentheses.

Affected versions

Root cause

Bisected to d9ace1f ("fix: resolve several empty line regressions", #330). Its parent 7798699 is clean.

That commit adds a Python < 3.12 workaround to classify.is_f_string:

elif any(
    [
        token.string.startswith('f"""'),
        prev_token.string.startswith('f"""'),
        token.string.startswith("f'''"),
        prev_token.string.startswith("f'''"),
    ]
):
    return True

The PY312 branch above it tests tokenize.FSTRING_MIDDLE, which only appears inside an f-string. The < 3.12 branch instead matches any token whose text begins with f""", including an ordinary triple-quoted f-string that is not a docstring — and, via prev_token, the token that follows one.

The single consumer is _get_unmatched_start_end_indices (format.py:499), where a true verdict sets _start_row = prev_token.end[0] (format.py:526-535). That rewrites the f-string's start row back to where the previous token ended, so untokenize sees a token starting above one it has already emitted and raises.

Impact

With several files on one command line, the run aborts at the first offending file and exits 1, leaving every file after it unprocessed. On a 1,255-file tree this stopped at file 318 with no indication that the remainder was skipped.

Environment

CPython 3.11.15, Linux x86-64, docformatter==1.7.8 from PyPI.

Start with classify.is_f_string and its consumer _get_unmatched_start_end_indices in format.py:499-535, then run the supplied multi-line f-string reproducer on Python 3.11. Trace how the pre-3.12 f-string check changes the token start row; done means the reproducer formats without ValueError and multi-file runs do not abort on this input.