python-poetry / python-poetry/tomlkit
toml-test datetime cases use tomlkit's own parse_rfc3339 as the expected value, so they cannot detect datetime parsing bugs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 850
- Forks
- 162
- Avg merge
- 13m
- Merged PRs (30d)
- 2
Description
While running a mutation-based check of the test suite at 4b38bec, one structural gap stood out and seemed worth reporting. Nothing here is a bug in tomlkit's code — every mutation below was a deliberate break, and the code as shipped is correct.
What was observed
tests/test_toml_tests.py builds the expected value for every datetime-typed case with tomlkit._utils.parse_rfc3339:
stypes = {
...
"datetime": parse_rfc3339,
"datetime-local": parse_rfc3339,
"date-local": parse_rfc3339,
"time-local": parse_rfc3339,
}
That is the function under test, so for those four types the parser and the expectation drift together. Breaking the negative-offset sign in parse_rfc3339 (if sign == "-": offset = -offset → never negating) produced:
| red | |
|---|---|
| toml-test compliance cases (680) | 0 |
| hand-written tests (378) | 3 (test_parse_rfc3339_datetime ×2, test_datetimes_behave_like_datetimes) |
By contrast, dropping the integer sign in the parser reddened 6 compliance cases and untranslated string escapes 7 — for int and str the corpus is an independent oracle. For datetimes it is not, and the entire guard is three hand-written tests.
Related, smaller: no input anywhere in tests/ (hand-written or corpus) carries a non-zero minute offset, so +05:30 being parsed as +05:00 (dropping minute_offset from the timedelta) survives the whole suite.
Possible fix
Build datetime expectations independently — datetime.fromisoformat accepts RFC 3339 on Python 3.11+ and the project already requires ≥3.9 with a compat shim, or a tiny hand-rolled regex as the toml-test JSON format is fixed — and add one +05:30-style case to test_parse_rfc3339_datetime.
Also noticed
tests/util.py::elementary_test asserts isinstance(v.unwrap(), unwrapped_type) only, so the nine test_*_unwrap tests detect a wrong type but not a wrong value: Bool.unwrap returning not bool(self) passes test_true_unwrap and test_false_unwrap. Passing the expected value to the helper would close that.
Full write-up with the mutation spec and per-test results, reproducible against 4b38bec: https://github.com/ArnauFerma/falsifiable-tests/blob/main/case-studies/tomlkit.md
Happy to send a PR for either if you'd take one.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/test_toml_tests.py and the parse_rfc3339 expectations, then run the datetime tests, including test_parse_rfc3339_datetime. Make datetime expectations independent of the parser and add coverage for a +05:30 offset; if taking the related helper issue, inspect tests/util.py::elementary_test and the test_*_unwrap cases, with value assertions as the completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100