python / python/cpython

Pure-Python `date.fromisoformat` silently mis-parses malformed basic-format dates

Aberta
#152,204 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

stdlib type-bug
Linguagem predominante
Python
Estrelas
77.2k
Forks
36k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

Bug description

_pydatetime.date.fromisoformat (the pure-Python reference used when the C
accelerator is unavailable, and via _pydatetime directly) returns a
wrong-but-plausible date for several strings that are not valid ISO-8601
dates. The C accelerator raises ValueError for every one of them. Because the
result is a silently incorrect date rather than an error, malformed input
becomes valid-looking data with no signal that anything went wrong.

There are two surface forms of the same underlying defect in
_parse_isoformat_date, which slices fixed-width substrings and calls int()
on them without checking that each slice is exactly N ASCII digits:

(1) int() tolerates a leading + / - / space in a basic-format field.

>>> import _datetime, _pydatetime
>>> _pydatetime.date.fromisoformat('2020+12')
datetime.date(2020, 1, 2)
>>> _datetime.date.fromisoformat('2020+12')
Traceback (most recent call last):
  ...
ValueError: Invalid isoformat string: '2020+12'
>>> _pydatetime.date.fromisoformat('+020-06-15')
datetime.date(20, 6, 15)
>>> _pydatetime.date.fromisoformat('2020-W 5')
datetime.date(2020, 1, 27)
>>> _pydatetime.date.fromisoformat('202012+9')
datetime.date(2020, 12, 9)
>>> _pydatetime.date.fromisoformat('2020 12')
datetime.date(2020, 1, 2)

Here int('+1') == 1, int(' 1') == 1 and int('+9') == 9, so the month /
day / week fields parse a sign or space that is not part of any ISO-8601 date.

(2) The length gate admits 7-character strings, and a fixed-width slice then
reads a 1-character tail.

>>> _pydatetime.date.fromisoformat('2020061')
datetime.date(2020, 6, 1)
>>> _datetime.date.fromisoformat('2020061')
Traceback (most recent call last):
  ...
ValueError: Invalid isoformat string: '2020061'
>>> _pydatetime.date.fromisoformat('2020123')
datetime.date(2020, 12, 3)
>>> _pydatetime.date.fromisoformat('2020-W2')
datetime.date(2020, 1, 6)
>>> _pydatetime.date.fromisoformat('9999121')
datetime.date(9999, 12, 1)

'2020061' is 7 chars; the gate len(date_string) in (7, 8, 10) lets it
through, the month slice reads '06' and the day slice dtstr[6:8] reads the
1-character tail '1', giving date(2020, 6, 1). '2020-W2' reads a 1-digit
week int('2'). The C parse_digits(p, ..., 2) requires exactly two digits, so
C rejects all of these.

datetime.fromisoformat inherits the same defect via the date branch, e.g.
_pydatetime.datetime.fromisoformat('2020061') returns
datetime.datetime(2020, 6, 1, 0, 0) while the C path raises.

C vs pure-Python
input C _datetime pure-Python _pydatetime
date.fromisoformat('2020+12') ValueError date(2020, 1, 2)
date.fromisoformat('+020-06-15') ValueError date(20, 6, 15)
date.fromisoformat('2020-W 5') ValueError date(2020, 1, 27)
date.fromisoformat('202012+9') ValueError date(2020, 12, 9)
date.fromisoformat('2020061') ValueError date(2020, 6, 1)
date.fromisoformat('2020123') ValueError date(2020, 12, 3)
date.fromisoformat('2020-W2') ValueError date(2020, 1, 6)
date.fromisoformat('9999121') ValueError date(9999, 12, 1)
Root cause

Lib/_pydatetime.py, _parse_isoformat_date (the function's own comment notes
it "assumes an ASCII-only string of lengths 7, 8 or 10"). On current main the
function body is:

def _parse_isoformat_date(dtstr):
    # It is assumed that this is an ASCII-only string of lengths 7, 8 or 10,
    # see the comment on Modules/_datetimemodule.c:_find_isoformat_datetime_separator
    if len(dtstr) not in (7, 8, 10):           # line 361
        raise ValueError("Invalid isoformat string")
    year = int(dtstr[0:4])                     # line 363
    ...
        weekno = int(dtstr[pos:pos + 2])       # line 370  (week field)
        ...
        dayno = int(dtstr[pos:pos + 1])        # line 380  (week day field)
    ...
        month = int(dtstr[pos:pos + 2])        # line 384  (month field)
        ...
        day = int(dtstr[pos:pos + 2])          # line 390  (day field)

The if len(dtstr) not in (7, 8, 10) gate at line 361 only bounds the total
length; date.fromisoformat (the caller, lines 1059-1060) applies the same
length gate before calling in. Neither gate checks the content of the
fixed-width fields. Each field is read with int(dtstr[pos:pos+N]). int()
accepts a leading +/-/whitespace and a short string, so:

  • a +/-/space that lands in a month/day/week field is silently consumed
    (form 1), and
  • on a 7-char string the day/week slice runs off the end and int() happily
    parses the 1-character remainder (form 2).

Wrong side: pure-Python, which over-accepts. ISO-8601 calendar dates contain
no sign or space inside the date, and have no 1-digit month-day or 1-digit
week; date.fromisoformat's docstring promises a string "in the format emitted
by date.isoformat()". The C accelerator's parse_digits rejects any non-digit
byte and requires the exact field width, then verifies the whole string was
consumed, so C is correct here.

Suggested fix

Validate each slice in _parse_isoformat_date before converting: require that
the year/month/day/week/weekday slice is exactly N ASCII digits (mirroring the
C parse_digits). The module already has an _is_ascii_digit helper used by
the fraction path (_parse_hh_mm_ss_ff does
all(map(_is_ascii_digit, tstr[pos:]))), so reusing it keeps the check
consistent, e.g. raise ValueError unless
len(s) == N and all(map(_is_ascii_digit, s)) before calling int(s). That
makes the malformed basic-format strings above raise ValueError on the
pure-Python path exactly as the C accelerator does, and closes the 7-char
short-slice hole at the same time. (The length gate (7, 8, 10) can stay; the
per-field width check is what rejects '2020061', since its day slice is then a
1-char string.)

Environment
  • Reproduced on current main, 3.16.0a0.
  • On current main, the length gate in _parse_isoformat_date is already an
    if len(dtstr) not in (7, 8, 10): raise ValueError (gh-152060 / PR #152061,
    merged, replaced the earlier assert len(dtstr) in (7, 8, 10) so that a bad
    length raises ValueError instead of AssertionError). That change only
    touched the length-gate exception type; it did not touch the
    int(dtstr[...]) slices, so both mis-parse forms above still reproduce on
    main. (A pre-PR-#152061 checkout still carries the assert; the slice
    defect is the same either way.)
  • The same slice-based _parse_isoformat_date exists in the 3.14 / 3.15
    branches (the pure-Python module is _pydatetime in all of them), so those
    branches are affected wherever the pure-Python path is exercised (the C
    accelerator masks it when present).
Relation to existing issues

This is distinct from the known nearby issues:

  • gh-107779 ("incorrectly accepts and parses strings without date-time
    separator", open; PR #107791) is about _find_isoformat_datetime_separator
    returning an index that is not a real separator in datetime.fromisoformat.
    That is a different function and mechanism; it does not address the
    int()-slice leniency in _parse_isoformat_date. Verified on this build that
    '2024-01-17T15:21:00-0800' (the basic/extended-mixing class) is accepted by
    both implementations, i.e. not a C-vs-pure-Python divergence.
  • gh-152060 / PR #152061 (merged) was the exception-type fix: a
    wrong-length dtstr used to raise AssertionError (from the old
    assert len(...)); PR #152061 turned that into a ValueError. That is a
    disjoint defect: it is about the length gate's exception type, whereas this
    issue is about strings of a valid length (7/8/10) whose fixed-width fields
    are mis-sliced into a silently wrong value. The per-field slices PR #152061
    left untouched are exactly the ones at fault here.

Found with a differential C-vs-pure-Python fromisoformat testing harness
(AI-assisted, each case hand-verified).

Linked PRs
  • gh-152205
  • gh-156359
  • gh-156367
  • gh-156368

Guia de contribuição

Abrir o guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Direção de pesquisa

Comece em Lib/_pydatetime.py, em _parse_isoformat_date, e compare o parsing de largura fixa com o comportamento de C parse_digits descrito na issue. Verifique as entradas malformadas no formato básico listadas e, em seguida, adicione validação da largura dos campos e dos dígitos ASCII para que os caminhos de date e datetime em Python puro gerem ValueError como _datetime.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
python
Domínio
backend
Tipo de issue
Bug
Dificuldade
3/5
Tempo estimado
1-2 dias
Status de atividade
Estagnada
Clareza
Claramente especificada
Facilidade para iniciantes
25/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.