python-poetry / python-poetry/tomlkit

toml-test datetime cases use tomlkit's own parse_rfc3339 as the expected value, so they cannot detect datetime parsing bugs

Open
#603 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
850
Forks
162
Avg merge
13m
Merged PRs (30d)
2

Description

While running a mutation-based check of the test suite at 4b38bec, one structural gap stood out and seemed worth reporting. Nothing here is a bug in tomlkit's code — every mutation below was a deliberate break, and the code as shipped is correct.

What was observed

tests/test_toml_tests.py builds the expected value for every datetime-typed case with tomlkit._utils.parse_rfc3339:

stypes = {
    ...
    "datetime": parse_rfc3339,
    "datetime-local": parse_rfc3339,
    "date-local": parse_rfc3339,
    "time-local": parse_rfc3339,
}

That is the function under test, so for those four types the parser and the expectation drift together. Breaking the negative-offset sign in parse_rfc3339 (if sign == "-": offset = -offset → never negating) produced:

red
toml-test compliance cases (680) 0
hand-written tests (378) 3 (test_parse_rfc3339_datetime ×2, test_datetimes_behave_like_datetimes)

By contrast, dropping the integer sign in the parser reddened 6 compliance cases and untranslated string escapes 7 — for int and str the corpus is an independent oracle. For datetimes it is not, and the entire guard is three hand-written tests.

Related, smaller: no input anywhere in tests/ (hand-written or corpus) carries a non-zero minute offset, so +05:30 being parsed as +05:00 (dropping minute_offset from the timedelta) survives the whole suite.

Possible fix

Build datetime expectations independently — datetime.fromisoformat accepts RFC 3339 on Python 3.11+ and the project already requires ≥3.9 with a compat shim, or a tiny hand-rolled regex as the toml-test JSON format is fixed — and add one +05:30-style case to test_parse_rfc3339_datetime.

Also noticed

tests/util.py::elementary_test asserts isinstance(v.unwrap(), unwrapped_type) only, so the nine test_*_unwrap tests detect a wrong type but not a wrong value: Bool.unwrap returning not bool(self) passes test_true_unwrap and test_false_unwrap. Passing the expected value to the helper would close that.

Full write-up with the mutation spec and per-test results, reproducible against 4b38bec: https://github.com/ArnauFerma/falsifiable-tests/blob/main/case-studies/tomlkit.md

Happy to send a PR for either if you'd take one.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/test_toml_tests.py and the parse_rfc3339 expectations, then run the datetime tests, including test_parse_rfc3339_datetime. Make datetime expectations independent of the parser and add coverage for a +05:30 offset; if taking the related helper issue, inspect tests/util.py::elementary_test and the test_*_unwrap cases, with value assertions as the completion criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.