ValueError: start precedes previous end, for a backslash line continuation after a reformatted docstring
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
docformatter 1.7.8 (and current master, d5c7b77) crashes with the tokenizer ValueError whenever a docstring edit shifts the lines of a file that contains a backslash line continuation somewhere after that docstring. Running docformatter --check over the Python 3.14 standard library hits it in _sitebuiltins.py, pdb.py, tokenize.py, ctypes/util.py, email/feedparser.py, encodings/utf_16.py, encodings/utf_32.py, encodings/utf_8_sig.py and http/cookies.py.
Reproducer
class A:
"""Doc."""
x = 1 \
+ 2
$ docformatter --check t.py
Traceback (most recent call last):
...
File ".../docformatter/format.py", line 919, in _do_format_code
_code = tokenize.untokenize(self.new_tokens)
File ".../tokenize.py", line 178, in add_whitespace
raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (4,8) precedes previous end (4,9)
A module docstring followed directly by x = 1 \ / + 2 fails the same way. Adding a blank line after the docstring (so nothing needs to be inserted) makes it pass, and so does writing the continuation with parentheses instead of a backslash. Same result on Python 3.11 and 3.14.
Cause
docformatter inserts the missing blank line after the docstring, which shifts every following token down one row. _do_update_token_indices() then recomputes positions and decides whether a token continues the previous token's row with
is_same_position = tokens[i].start[0] == tokens[i - 1].end[0]
where tokens[i - 1] has already been shifted and tokens[i] has not. For a parenthesised continuation an NL token sits between the two physical lines, so the comparison is never reached; a backslash continuation produces no NL token, so for the first token on the continuation line the un-shifted row happens to equal the shifted row of the previous token, the token is treated as being on the same row, and its start ((4,8)) ends up before the previous token's end ((4,9)).
I have a fix (treat a token as starting a new row when the previous token's physical line ends with \) with a regression test; PR follows.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docformatter/format.py, especially _do_format_code and _do_update_token_indices, then run the provided t.py reproducer with --check. Add or inspect the regression test mentioned in the issue and verify backslash continuations after reformatted docstrings no longer raise the tokenizer ValueError on the supported Python versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100