Version dependence in the test suite

オープン

#2,069 2025年8月16日に作成コメント 0 件リアクション 0 件担当者 0 名 GitHub で見る

まだ誰も着手していません。

topic: other

主要言語: Python
スター: 1.8k
フォーク: 302
平均マージ: 23時間
マージ済み PR（30日）: 8

説明

While working on #2068 I ran the conformance test suite under Python 3.14 and got a number of changes:

diff --git a/conformance/results/mypy/classes_classvar.toml b/conformance/results/mypy/classes_classvar.toml
index 0072d63..ce30b65 100644
--- a/conformance/results/mypy/classes_classvar.toml
+++ b/conformance/results/mypy/classes_classvar.toml
@@ -12,7 +12,6 @@ classes_classvar.py:40: error: Name "var" is not defined  [name-defined]
 classes_classvar.py:45: error: ClassVar cannot contain type variables  [misc]
 classes_classvar.py:46: error: ClassVar cannot contain type variables  [misc]
 classes_classvar.py:52: error: Incompatible types in assignment (expression has type "dict[Never, Never]", variable has type "list[str]")  [assignment]
-classes_classvar.py:54: error: Variable should not be annotated with both ClassVar and Final  [misc]
 classes_classvar.py:55: error: Invalid type: ClassVar nested inside other type  [valid-type]
 classes_classvar.py:67: error: Invalid type: ClassVar nested inside other type  [valid-type]
 classes_classvar.py:69: error: ClassVar can only be used for assignments in class body  [misc]
@@ -31,5 +30,6 @@ classes_classvar.py:140: note: Protocol member ProtoA.y expected class variable,
 conformance_automated = "Fail"
 errors_diff = """
 Line 47: Expected 1 errors
+Line 54: Expected 1 errors
 Line 67: Unexpected errors ['classes_classvar.py:67: error: Invalid type: ClassVar nested inside other type  [valid-type]']
 """
diff --git a/conformance/results/mypy/dataclasses_final.toml b/conformance/results/mypy/dataclasses_final.toml
index fdf8691..eb91705 100644
--- a/conformance/results/mypy/dataclasses_final.toml
+++ b/conformance/results/mypy/dataclasses_final.toml
@@ -7,12 +7,10 @@ conformance_automated = "Fail"
 errors_diff = """
 Line 27: Expected 1 errors
 Line 16: Unexpected errors ['dataclasses_final.py:16: error: Final name must be initialized with a value  [misc]']
-Line 18: Unexpected errors ['dataclasses_final.py:18: error: Final can be only used as an outermost qualifier in a variable annotation  [valid-type]']
 Line 24: Unexpected errors ['dataclasses_final.py:24: error: Expression is of type "Any", not "int"  [assert-type]']
 """
 output = """
 dataclasses_final.py:16: error: Final name must be initialized with a value  [misc]
-dataclasses_final.py:18: error: Final can be only used as an outermost qualifier in a variable annotation  [valid-type]
 dataclasses_final.py:24: error: Expression is of type "Any", not "int"  [assert-type]
 dataclasses_final.py:35: error: Cannot assign to final attribute "final_no_default"  [misc]
 dataclasses_final.py:36: error: Cannot assign to final attribute "final_with_default"  [misc]
diff --git a/conformance/results/mypy/generics_variance_inference.toml b/conformance/results/mypy/generics_variance_inference.toml
index b1158c7..ef877e1 100644
--- a/conformance/results/mypy/generics_variance_inference.toml
+++ b/conformance/results/mypy/generics_variance_inference.toml
@@ -6,6 +6,7 @@ generics_variance_inference.py:28: error: Incompatible types in assignment (expr
 generics_variance_inference.py:41: error: Incompatible types in assignment (expression has type "ShouldBeCovariant1[float]", variable has type "ShouldBeCovariant1[int]")  [assignment]
 generics_variance_inference.py:49: error: Incompatible types in assignment (expression has type "ShouldBeCovariant2[float]", variable has type "ShouldBeCovariant2[int]")  [assignment]
 generics_variance_inference.py:58: error: Incompatible types in assignment (expression has type "ShouldBeCovariant3[float]", variable has type "ShouldBeCovariant3[int]")  [assignment]
+generics_variance_inference.py:66: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[int]", variable has type "ShouldBeCovariant4[float]")  [assignment]
 generics_variance_inference.py:67: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[float]", variable has type "ShouldBeCovariant4[int]")  [assignment]
 generics_variance_inference.py:80: error: Incompatible types in assignment (expression has type "ShouldBeCovariant5[float]", variable has type "ShouldBeCovariant5[int]")  [assignment]
 generics_variance_inference.py:96: error: Incompatible types in assignment (expression has type "ShouldBeInvariant1[int]", variable has type "ShouldBeInvariant1[float]")  [assignment]
@@ -24,6 +25,7 @@ generics_variance_inference.py:170: error: Incompatible types in assignment (exp
 generics_variance_inference.py:181: error: Incompatible types in assignment (expression has type "ShouldBeCovariant6[float]", variable has type "ShouldBeCovariant6[int]")  [assignment]
 generics_variance_inference.py:194: error: Incompatible types in assignment (expression has type "ShouldBeContravariant2[int]", variable has type "ShouldBeContravariant2[float]")  [assignment]
 """
-conformance_automated = "Pass"
+conformance_automated = "Fail"
 errors_diff = """
+Line 66: Unexpected errors ['generics_variance_inference.py:66: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[int]", variable has type "ShouldBeCovariant4[float]")  [assignment]']
 """
diff --git a/conformance/results/mypy/qualifiers_final_annotation.toml b/conformance/results/mypy/qualifiers_final_annotation.toml
index 2d0cea3..8c0830f 100644
--- a/conformance/results/mypy/qualifiers_final_annotation.toml
+++ b/conformance/results/mypy/qualifiers_final_annotation.toml
@@ -20,8 +20,6 @@ qualifiers_final_annotation.py:71: error: Cannot assign to final name "RATE"  [m
 qualifiers_final_annotation.py:81: error: Cannot assign to final attribute "DEFAULT_ID"  [misc]
 qualifiers_final_annotation.py:94: error: Cannot assign to final name "BORDER_WIDTH"  [misc]
 qualifiers_final_annotation.py:96: error: Cannot assign to final name "__private"  [misc]
-qualifiers_final_annotation.py:107: error: Final can be only used as an outermost qualifier in a variable annotation  [valid-type]
-qualifiers_final_annotation.py:108: error: Variable should not be annotated with both ClassVar and Final  [misc]
 qualifiers_final_annotation.py:118: error: Final can be only used as an outermost qualifier in a variable annotation  [valid-type]
 qualifiers_final_annotation.py:121: error: Final can be only used as an outermost qualifier in a variable annotation  [valid-type]
 qualifiers_final_annotation.py:131: error: Invalid "NamedTuple()" field name  [misc]
@@ -40,6 +38,8 @@ qualifiers_final_annotation.py:170: error: Cannot assign to final name "PI"  [mi
 """
 conformance_automated = "Fail"
 errors_diff = """
+Line 107: Expected 1 errors
+Line 108: Expected 1 errors
 Line 149: Expected 1 errors
 Line 59: Unexpected errors ['qualifiers_final_annotation.py:59: error: Cannot assign to final attribute "ID6"  [misc]']
 Line 96: Unexpected errors ['qualifiers_final_annotation.py:96: error: Cannot assign to final name "__private"  [misc]']
diff --git a/conformance/results/pyright/annotations_forward_refs.toml b/conformance/results/pyright/annotations_forward_refs.toml
index ca35b5b..b91c665 100644
--- a/conformance/results/pyright/annotations_forward_refs.toml
+++ b/conformance/results/pyright/annotations_forward_refs.toml
@@ -1,7 +1,5 @@
 conformant = "Pass"
 output = """
-annotations_forward_refs.py:22:7 - error: "ClassA" is not defined (reportUndefinedVariable)
-annotations_forward_refs.py:23:12 - error: "ClassA" is not defined (reportUndefinedVariable)
 annotations_forward_refs.py:24:7 - error: Union syntax cannot be used with string operand; use quotes around entire expression (reportGeneralTypeIssues)
 annotations_forward_refs.py:25:13 - error: Union syntax cannot be used with string operand; use quotes around entire expression (reportGeneralTypeIssues)
 annotations_forward_refs.py:41:10 - error: Call expression not allowed in type expression (reportInvalidTypeForm)
@@ -29,11 +27,13 @@ annotations_forward_refs.py:52:11 - error: Unary operator not allowed in type ex
 annotations_forward_refs.py:53:11 - error: Binary operator not allowed in type expression (reportInvalidTypeForm)
 annotations_forward_refs.py:54:11 - error: Type expressions cannot use format string literals (f-strings) (reportGeneralTypeIssues)
 annotations_forward_refs.py:55:10 - error: Module cannot be used as a type (reportGeneralTypeIssues)
-annotations_forward_refs.py:66:26 - error: "ClassB" is not defined (reportUndefinedVariable)
 annotations_forward_refs.py:80:14 - error: Type of "ClassF" could not be determined because it refers to itself (reportGeneralTypeIssues)
 annotations_forward_refs.py:80:14 - error: Variable not allowed in type expression (reportInvalidTypeForm)
-annotations_forward_refs.py:89:8 - error: Expected class but received "(self: Self@ClassD) -> None" (reportGeneralTypeIssues)
 """
-conformance_automated = "Pass"
+conformance_automated = "Fail"
 errors_diff = """
+Line 22: Expected 1 errors
+Line 23: Expected 1 errors
+Line 66: Expected 1 errors
+Line 89: Expected 1 errors
 """
diff --git a/conformance/results/pyright/generics_variance_inference.toml b/conformance/results/pyright/generics_variance_inference.toml
index 9a378ad..5c0f32e 100644
--- a/conformance/results/pyright/generics_variance_inference.toml
+++ b/conformance/results/pyright/generics_variance_inference.toml
@@ -23,10 +23,12 @@ generics_variance_inference.py:58:35 - error: Type "ShouldBeCovariant3[float]" i
   "ShouldBeCovariant3[float]" is not assignable to "ShouldBeCovariant3[int]"
     Type parameter "T@ShouldBeCovariant3" is covariant, but "float" is not a subtype of "int"
       "float" is not assignable to "int" (reportAssignmentType)
+generics_variance_inference.py:66:36 - error: Type "ShouldBeCovariant4[int]" is not assignable to declared type "ShouldBeCovariant4[float]"
+  "ShouldBeCovariant4[int]" is not assignable to "ShouldBeCovariant4[float]"
+    Type parameter "T@ShouldBeCovariant4" is invariant, but "int" is not the same as "float" (reportAssignmentType)
 generics_variance_inference.py:67:34 - error: Type "ShouldBeCovariant4[float]" is not assignable to declared type "ShouldBeCovariant4[int]"
   "ShouldBeCovariant4[float]" is not assignable to "ShouldBeCovariant4[int]"
-    Type parameter "T@ShouldBeCovariant4" is covariant, but "float" is not a subtype of "int"
-      "float" is not assignable to "int" (reportAssignmentType)
+    Type parameter "T@ShouldBeCovariant4" is invariant, but "float" is not the same as "int" (reportAssignmentType)
 generics_variance_inference.py:80:34 - error: Type "ShouldBeCovariant5[float]" is not assignable to declared type "ShouldBeCovariant5[int]"
   "ShouldBeCovariant5[float]" is not assignable to "ShouldBeCovariant5[int]"
     Type parameter "T@ShouldBeCovariant5" is covariant, but "float" is not a subtype of "int"
@@ -80,6 +82,7 @@ generics_variance_inference.py:194:37 - error: Type "ShouldBeContravariant2[int]
     Type parameter "T@ShouldBeContravariant2" is contravariant, but "int" is not a supertype of "float"
       "float" is not assignable to "int" (reportAssignmentType)
 """
-conformance_automated = "Pass"
+conformance_automated = "Fail"
 errors_diff = """
+Line 66: Unexpected errors ['generics_variance_inference.py:66:36 - error: Type "ShouldBeCovariant4[int]" is not assignable to declared type "ShouldBeCovariant4[float]"']
 """

Analysis:

In classes_classvar.py, mypy allows Final[ClassVar[int]] only in 3.13+. This is in line with the runtime behavior, as this construct throws a runtime error in 3.12 and earlier. Interestingly, mypy also disallows this when from __future__ import annotations is used (so there wouldn't be a runtime error). Mypy seems to be in the wrong here according to the spec: we still explicitly disallow Final + ClassVar except in dataclasses, even if it works at runtime.
qualifiers_final_annotation.py also has mypy allowing a Final/ClassVar it shouldn't.
dataclasses_final.py is a similar issue with mypy, but here it's a positive; it correctly allows Final + ClassVar in a dataclass.
In generics_variance_inference.py, both mypy and pyright now infer a generic frozen dataclass as invariant. This is due to https://discuss.python.org/t/make-replace-stop-interfering-with-variance-inference/96092 .
In annotations_forward_refs.py, pyright changes its behavior regarding names in annotations. This is mostly correct and in line with PEP 649, but one of the new errors (the last line) seems wrong to me.

Regardless of the specifics of these changes, the general lesson is that the behavior of the test suite depends on the Python version, and we need to make sure we use a consistent Python version to run the test suite (presumably, the most recent stable version, so currently 3.13). Optionally, we could also extend the framework to support checking some tests under a more specific Python version, so we can test version-dependent behavior.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

はじめの一歩

issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
リポジトリをフォークし、ブランチを切って変更します。
issue 番号を参照したプルリクエストを送ります。

調査の方向性

Python 3.14 で適合性テストスイートを実行し、conformance/results/mypy/ と conformance/results/pyright/ の結果を比較します。classes_classvar.py、dataclasses_final.py、generics_variance_inference.py、qualifiers_final_annotation.py、annotations_forward_refs.py を含む参照されたケースを読み、期待される結果でサポート対象バージョンの違いをどのように考慮すべきかを判断します。スイートのバージョン依存の結果が意図的かつ一貫して表現されていれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック: python
領域: testing-qa
issue の種類: バグ
難易度: 4/5
見積もり時間: 3〜5日
活発さ: 停滞
明瞭さ: 説明が足りない
初心者へのやさしさ: 35/100

Version dependence in the test suite

説明

コントリビューションガイド

はじめの一歩

調査の方向性

評価

新しい issue をメールで受け取る