PyCQA / PyCQA/docformatter

ValueError: start precedes previous end, for a backslash line continuation after a reformatted docstring

Open
#377 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

fresh
Dominant language
Python
Stars
598
Forks
93
PR merge metrics
No merged PRs in 30d

Description

docformatter 1.7.8 (and current master, d5c7b77) crashes with the tokenizer ValueError whenever a docstring edit shifts the lines of a file that contains a backslash line continuation somewhere after that docstring. Running docformatter --check over the Python 3.14 standard library hits it in _sitebuiltins.py, pdb.py, tokenize.py, ctypes/util.py, email/feedparser.py, encodings/utf_16.py, encodings/utf_32.py, encodings/utf_8_sig.py and http/cookies.py.

Reproducer

class A:
    """Doc."""
    x = 1 \
        + 2
$ docformatter --check t.py
Traceback (most recent call last):
  ...
  File ".../docformatter/format.py", line 919, in _do_format_code
    _code = tokenize.untokenize(self.new_tokens)
  File ".../tokenize.py", line 178, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (4,8) precedes previous end (4,9)

A module docstring followed directly by x = 1 \ / + 2 fails the same way. Adding a blank line after the docstring (so nothing needs to be inserted) makes it pass, and so does writing the continuation with parentheses instead of a backslash. Same result on Python 3.11 and 3.14.

Cause

docformatter inserts the missing blank line after the docstring, which shifts every following token down one row. _do_update_token_indices() then recomputes positions and decides whether a token continues the previous token's row with

is_same_position = tokens[i].start[0] == tokens[i - 1].end[0]

where tokens[i - 1] has already been shifted and tokens[i] has not. For a parenthesised continuation an NL token sits between the two physical lines, so the comparison is never reached; a backslash continuation produces no NL token, so for the first token on the continuation line the un-shifted row happens to equal the shifted row of the previous token, the token is treated as being on the same row, and its start ((4,8)) ends up before the previous token's end ((4,9)).

I have a fix (treat a token as starting a new row when the previous token's physical line ends with \) with a regression test; PR follows.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docformatter/format.py, especially _do_format_code and _do_update_token_indices, then run the provided t.py reproducer with --check. Add or inspect the regression test mentioned in the issue and verify backslash continuations after reformatted docstrings no longer raise the tokenizer ValueError on the supported Python versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.