python / python/cpython

Pure-Python `date.fromisoformat` silently mis-parses malformed basic-format dates

Đang mở
#152,204 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

stdlib type-bug
Ngôn ngữ chính
Python
Star
77.2k
Fork
35.9k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

Bug description

_pydatetime.date.fromisoformat (the pure-Python reference used when the C
accelerator is unavailable, and via _pydatetime directly) returns a
wrong-but-plausible date for several strings that are not valid ISO-8601
dates. The C accelerator raises ValueError for every one of them. Because the
result is a silently incorrect date rather than an error, malformed input
becomes valid-looking data with no signal that anything went wrong.

There are two surface forms of the same underlying defect in
_parse_isoformat_date, which slices fixed-width substrings and calls int()
on them without checking that each slice is exactly N ASCII digits:

(1) int() tolerates a leading + / - / space in a basic-format field.

>>> import _datetime, _pydatetime
>>> _pydatetime.date.fromisoformat('2020+12')
datetime.date(2020, 1, 2)
>>> _datetime.date.fromisoformat('2020+12')
Traceback (most recent call last):
  ...
ValueError: Invalid isoformat string: '2020+12'
>>> _pydatetime.date.fromisoformat('+020-06-15')
datetime.date(20, 6, 15)
>>> _pydatetime.date.fromisoformat('2020-W 5')
datetime.date(2020, 1, 27)
>>> _pydatetime.date.fromisoformat('202012+9')
datetime.date(2020, 12, 9)
>>> _pydatetime.date.fromisoformat('2020 12')
datetime.date(2020, 1, 2)

Here int('+1') == 1, int(' 1') == 1 and int('+9') == 9, so the month /
day / week fields parse a sign or space that is not part of any ISO-8601 date.

(2) The length gate admits 7-character strings, and a fixed-width slice then
reads a 1-character tail.

>>> _pydatetime.date.fromisoformat('2020061')
datetime.date(2020, 6, 1)
>>> _datetime.date.fromisoformat('2020061')
Traceback (most recent call last):
  ...
ValueError: Invalid isoformat string: '2020061'
>>> _pydatetime.date.fromisoformat('2020123')
datetime.date(2020, 12, 3)
>>> _pydatetime.date.fromisoformat('2020-W2')
datetime.date(2020, 1, 6)
>>> _pydatetime.date.fromisoformat('9999121')
datetime.date(9999, 12, 1)

'2020061' is 7 chars; the gate len(date_string) in (7, 8, 10) lets it
through, the month slice reads '06' and the day slice dtstr[6:8] reads the
1-character tail '1', giving date(2020, 6, 1). '2020-W2' reads a 1-digit
week int('2'). The C parse_digits(p, ..., 2) requires exactly two digits, so
C rejects all of these.

datetime.fromisoformat inherits the same defect via the date branch, e.g.
_pydatetime.datetime.fromisoformat('2020061') returns
datetime.datetime(2020, 6, 1, 0, 0) while the C path raises.

C vs pure-Python
input C _datetime pure-Python _pydatetime
date.fromisoformat('2020+12') ValueError date(2020, 1, 2)
date.fromisoformat('+020-06-15') ValueError date(20, 6, 15)
date.fromisoformat('2020-W 5') ValueError date(2020, 1, 27)
date.fromisoformat('202012+9') ValueError date(2020, 12, 9)
date.fromisoformat('2020061') ValueError date(2020, 6, 1)
date.fromisoformat('2020123') ValueError date(2020, 12, 3)
date.fromisoformat('2020-W2') ValueError date(2020, 1, 6)
date.fromisoformat('9999121') ValueError date(9999, 12, 1)
Root cause

Lib/_pydatetime.py, _parse_isoformat_date (the function's own comment notes
it "assumes an ASCII-only string of lengths 7, 8 or 10"). On current main the
function body is:

def _parse_isoformat_date(dtstr):
    # It is assumed that this is an ASCII-only string of lengths 7, 8 or 10,
    # see the comment on Modules/_datetimemodule.c:_find_isoformat_datetime_separator
    if len(dtstr) not in (7, 8, 10):           # line 361
        raise ValueError("Invalid isoformat string")
    year = int(dtstr[0:4])                     # line 363
    ...
        weekno = int(dtstr[pos:pos + 2])       # line 370  (week field)
        ...
        dayno = int(dtstr[pos:pos + 1])        # line 380  (week day field)
    ...
        month = int(dtstr[pos:pos + 2])        # line 384  (month field)
        ...
        day = int(dtstr[pos:pos + 2])          # line 390  (day field)

The if len(dtstr) not in (7, 8, 10) gate at line 361 only bounds the total
length; date.fromisoformat (the caller, lines 1059-1060) applies the same
length gate before calling in. Neither gate checks the content of the
fixed-width fields. Each field is read with int(dtstr[pos:pos+N]). int()
accepts a leading +/-/whitespace and a short string, so:

  • a +/-/space that lands in a month/day/week field is silently consumed
    (form 1), and
  • on a 7-char string the day/week slice runs off the end and int() happily
    parses the 1-character remainder (form 2).

Wrong side: pure-Python, which over-accepts. ISO-8601 calendar dates contain
no sign or space inside the date, and have no 1-digit month-day or 1-digit
week; date.fromisoformat's docstring promises a string "in the format emitted
by date.isoformat()". The C accelerator's parse_digits rejects any non-digit
byte and requires the exact field width, then verifies the whole string was
consumed, so C is correct here.

Suggested fix

Validate each slice in _parse_isoformat_date before converting: require that
the year/month/day/week/weekday slice is exactly N ASCII digits (mirroring the
C parse_digits). The module already has an _is_ascii_digit helper used by
the fraction path (_parse_hh_mm_ss_ff does
all(map(_is_ascii_digit, tstr[pos:]))), so reusing it keeps the check
consistent, e.g. raise ValueError unless
len(s) == N and all(map(_is_ascii_digit, s)) before calling int(s). That
makes the malformed basic-format strings above raise ValueError on the
pure-Python path exactly as the C accelerator does, and closes the 7-char
short-slice hole at the same time. (The length gate (7, 8, 10) can stay; the
per-field width check is what rejects '2020061', since its day slice is then a
1-char string.)

Environment
  • Reproduced on current main, 3.16.0a0.
  • On current main, the length gate in _parse_isoformat_date is already an
    if len(dtstr) not in (7, 8, 10): raise ValueError (gh-152060 / PR #152061,
    merged, replaced the earlier assert len(dtstr) in (7, 8, 10) so that a bad
    length raises ValueError instead of AssertionError). That change only
    touched the length-gate exception type; it did not touch the
    int(dtstr[...]) slices, so both mis-parse forms above still reproduce on
    main. (A pre-PR-#152061 checkout still carries the assert; the slice
    defect is the same either way.)
  • The same slice-based _parse_isoformat_date exists in the 3.14 / 3.15
    branches (the pure-Python module is _pydatetime in all of them), so those
    branches are affected wherever the pure-Python path is exercised (the C
    accelerator masks it when present).
Relation to existing issues

This is distinct from the known nearby issues:

  • gh-107779 ("incorrectly accepts and parses strings without date-time
    separator", open; PR #107791) is about _find_isoformat_datetime_separator
    returning an index that is not a real separator in datetime.fromisoformat.
    That is a different function and mechanism; it does not address the
    int()-slice leniency in _parse_isoformat_date. Verified on this build that
    '2024-01-17T15:21:00-0800' (the basic/extended-mixing class) is accepted by
    both implementations, i.e. not a C-vs-pure-Python divergence.
  • gh-152060 / PR #152061 (merged) was the exception-type fix: a
    wrong-length dtstr used to raise AssertionError (from the old
    assert len(...)); PR #152061 turned that into a ValueError. That is a
    disjoint defect: it is about the length gate's exception type, whereas this
    issue is about strings of a valid length (7/8/10) whose fixed-width fields
    are mis-sliced into a silently wrong value. The per-field slices PR #152061
    left untouched are exactly the ones at fault here.

Found with a differential C-vs-pure-Python fromisoformat testing harness
(AI-assisted, each case hand-verified).

Linked PRs
  • gh-152205
  • gh-156359
  • gh-156367
  • gh-156368

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Hướng nghiên cứu

Bắt đầu trong Lib/_pydatetime.py tại _parse_isoformat_date và so sánh cách phân tích cú pháp độ rộng cố định của nó với hành vi của C parse_digits được mô tả trong issue. Kiểm tra các đầu vào basic-format không hợp lệ được liệt kê, sau đó thêm kiểm tra độ rộng trường và chữ số ASCII để các đường dẫn date và datetime thuần Python phát sinh ValueError giống như _datetime.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
backend
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
25/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.