dotnet / dotnet/winforms

Unstable unit tests that need to be disabled and fixed

Open
#11,227 23 comments 0 reactions 1 assignee Claimed by @LeafShi1 View on GitHub
Dominant language
C#
Stars
4.9k
Forks
1.1k
Avg merge
1d 13m
Merged PRs (30d)
85

Description

**Disable failing tests and open tracking bugs**

1. When test fails in the CI or PR build, open a bug for each test fail to investigate the test further and add a `disabled-test` label to the bug.
2. Disable test by adding a [skip] attribute, If only a specific theory data causes the failure, comment that data out. (In order to prevent some problems from being caused by changes in the PR itself or machine problems, if it fails only once in a month, we will not disable it. If it fails a second time, we will disable it.)
4. Add a link to the tracking bug to the SkipAttribute of the comment line.
5. Collect all data relevant to the failure into the bug.
6. Try to root-cause the flakiness, and fix these tests:
- Replacing SendInput or cursor-positioning steps with direct method invocations, or by sending WM_COMMAND
- If UI element is easily identifiable, for example by and accessible name, or we have access to the AccessibleObject, we can use UIA
APIs to invoke methods
- Add logging if possible.
- If the test must use `SendInput`, then make sure it is moved to the UI Integration tests assembly

**The following unit tests exist flaky issues**:

- [X] System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_Item_Get32bppColorDepth_Success(pixelFormat: Format1bppIndexed, pixel00Color: Color [Empty], givenPixel10Color: Color [Empty], expectedPixel10Color: Color [A=255, R=0, G=0, B=0])
- [X] System.Windows.Forms.Tests.ToolTipTests.ToolTip_SetToolTip_TabControl_DoesNotAddToolForTabControlItself
- [X] System.Windows.Forms.Tests.ToolTipTests.ToolTip_WmShow_Invokes_AnnounceText_WithExpectedText_ForTabControlTabs
- [x] System.Windows.Forms.Tests.DataGridViewTests.DataGridView_OnColumnHeadersHeightChanged_InvokeWithHandle_CallsColumnHeadersHeightChanged(columnHeadersWidthSizeMode: EnableResizing, columnHeadersVisible: True, eventArgs: null)
- [x] System.Windows.Forms.UITests.ButtonTests.Button_CancelButton_EscapeClicksCancelButtonAsync
- [x] System.Windows.Forms.UITests.ButtonTests.Button_DialogResult_SpaceToClickFocusedButtonAsync
- [x] System.Windows.Forms.UITests.ButtonTests.Button_Hotkey_Fires_OnClickAsync
- [x] System.Windows.Forms.UITests.ButtonTests.Button_Press_Enter_Fires_OnClickAsync
- [x] System.Windows.Forms.UITests.ListViewTests.ListView_Group_NavigateKeyboard_SucceedsAsync
- [x] System.Windows.Forms.UITests.NumericUpDownTests.NumericUpDownAccessibleObject_Focused_ReturnsCorrectValueAsync
- [x] System.Windows.Forms.Tests.CheckBoxRendererTests.CheckBoxRenderer_DrawCheckBox_OverloadWithSizeAndText
- [x] System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_ClearUndo_CanUndo_Success
- [x] System.Windows.Forms.Tests.ToolStripTests.ToolStrip_WndProc_InvokeMouseActivate_Success
- [x] System.Windows.Forms.Tests.ToolStripTests.ToolStrip_WndProc_InvokeMouseActivateWithHandle_Success
- [ ] System.Windows.Forms.Tests.DataGridViewTests.DataGridView_ColumnHeadersHeight_SetWithHandle_GetReturnsExpected(columnHeadersWidthSizeMode: AutoSize, columnHeadersVisible: True, autoSize: True, value: 4, expectedValue: 18, expectedInvalidatedCallCount: 0)
- [ ] System.Windows.Forms.Tests.DataGridViewTests.DataGridView_ColumnHeadersHeight_SetWithParentWithHandle_GetReturnsExpected(columnHeadersWidthSizeMode: AutoSize, columnHeadersVisible: True, autoSize: True, value: 4, expectedValue: 18, expectedInvalidatedCallC
- [x] System.Windows.Forms.Tests.MenuStripTests.MenuStrip_WndProc_InvokeMouseActivate_Success
- [x] System.Windows.Forms.Tests.MenuStripTests.MenuStrip_WndProc_InvokeMouseActivateWithHandle_Success
- [x] System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Copy_PasteNotEmptyWithHandle_Success
- [ ] System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Paste_InvokeNotEmpty_Success
- [x] System.Windows.Forms.Tests.ListViewItem_IKeyboardToolTipTests.ListViewItemKeyboardToolTip_InvokeIsHoveredWithMouse_ReturnsExpected(insideListView: True, virtualMode: True, isHovered: True, expected: True)
- [x] System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Undo_CanUndo_Success
- [x] System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Copy_PasteNotEmpty_Success
- [x] System.Windows.Forms.Tests.BindingContextTests.BindingContext_Item_GetIListSourceDataSourceWithDataMemberReturningIListNull_ThrowsArgumentNullException
- [ ] System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetText_InvokeString_GetReturnsExpected
- [x] System.Windows.Forms.Tests.TreeNode_TreeNodeIKeyboardToolTipTests.TreeNodeIKeyboardToolTip_InvokeIsHoveredWithMouse_ReturnsExpected(insideTreeView: True, isHovered: True, expected: True)
- [ ] System.Windows.Forms.Tests.TextBoxBaseTests+ClipboardTests.TextBoxBase_Copy_PasteNotEmptyWithHandle_Success
- [ ] System.Windows.Forms.Tests.TreeViewAccessibilityObjectTests.TreeViewAccessibilityObject_Role_IsOutline_ByDefault
- [ ] System.Windows.Forms.Tests.TextBoxBaseTests+ClipboardTests.TextBoxBase_Undo_CanUndo_Success
- [ ] Microsoft.VisualBasic.Devices.Tests.NetworkTests.PingUri_ShortTimeout_Success
- [ ] System.Windows.Forms.Tests.AccessibleObjects.TextBoxAccessibleObjectTests.TextBoxAccessibilityObject_Role_IsText_ByDefault
- [ ] System.Windows.Forms.Tests.CheckBoxRendererTests.CheckBoxRenderer_DrawCheckBox_VisualStyleOn_OverloadWithTextFormat
- [ ] System.Windows.Forms.UITests.MonthCalendarTests.MonthCalendar_Click_Date_InvokeEventsAsync(delta: -1)
- [ ] Microsoft.VisualBasic.Forms.Tests.WindowsFormsApplicationBaseTests.Run_SingleInstanceNoStartupFormException
- [x] System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetText_InvokeStringTextDataFormat_GetReturnsExpected(format: Text)
- [ ] System.Private.Windows.Core.BinaryFormat.Tests.WinFormsBinaryFormattedObjectTests.BinaryFormattedObject_ImageListStreamer_FromBinaryFormatter
- [ ] System.Windows.Forms.Tests.RadioButtonRendererTests.RadioButtonRenderer_DrawRadioButton_OverloadWithHandle
- [ ] System.Windows.Forms.Tests.ClipboardTests.Clipboard_RoundTrip_ManagedDataObject_SupportsTypedInterface
- [ ] System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetDataObject_InvokeObjectIComDataObject_GetReturnsExpected(data: 1)
- [ ] System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetDataObject_WithJson_ReturnsExpected
- [ ] System.Windows.Forms.Tests.ClipboardTests.RoundTrip_DataObject_SupportsTypedInterface
- [ ] System.Windows.Forms.Tests.ClipboardTests.SetDataObject_InvokeObjectBoolIComDataObject_GetReturnsExpected
- [ ] System.Windows.Forms.Tests.ClipboardTests.SetDataObject_InvokeObjectBoolNotIComDataObject_GetReturnsExpected
- [x] System.Windows.Forms.Design.Tests.DesignerAttributeTests.EditorAttribute_TypeExists
- [ ] System.Windows.Forms.Tests.ClipboardTests.SetDataObject_WithJson_ReturnsExpected
- [x] System.Windows.Forms.UITests.ListViewTests.ListView_Click_On_Second_Column_Does_Not_Alter_CheckBoxesAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_FragmentNavigate_ReturnsNullAsync(columnCount: 1, subItemsCount: 0, width: 150, height: 150, direction: 3)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_FragmentNavigate_WorksExpectedAsync(columnCount: 2, subItemsCount: 2, width: 150, height: 150, itemIndex: 0, subItemIndex: 1, direction: 3)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsExpected_For_BigSizeAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsExpectedAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsNullAsync(columnCount: 4, subItemsCount: 0, width: 150, height: 150)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildAsync(columnCount: 4, subItemsCount: 1)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildCount_ReturnsExpectedAsync(columnCount: 4, subItemsCount: 3, width: 150, height: 150, expected: 3)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ForDifferentSize_ReturnsExpectedAsync(columnCount: 4, subItemsCount: 1, width: 150, height: 150, expected1: -1, expected2: 1)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpected_For_BigSizeAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpected_SubItemsMoreThanColumnsAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpectedAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Child_ReturnsNullAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_NextSibling_ReturnsExpectedAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Sibling_ReturnsNull_For_SmallSizeAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Sibling_ReturnsNullAsync(columnCount: 4, subItemsCount: 1)
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_HitTest_ReturnExpectedAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_ColumnProperty_ReturnsMinusOneAsync
- [ ] System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsMinusOne_For_Single_ColumnAsync
- [ ] System.Windows.Forms.Tests.ClipboardTests.SetDataObject_WithMultipleData_ReturnsExpected
- [ ] System.Windows.Forms.Tests.HtmlDocumentTests.HtmlDocument_LinkColor_GetCustomValueSet_ReturnsExpected(value: null, expected: Color [A=255, R=0, G=0, B=255])
- [ ] System.Windows.Forms.Analyzers.Tests.AvoidPassingTaskWithoutCancellationTokenTests.CS_AvoidPassingTaskWithoutCancellationAnalyzer
- [ ] System.Windows.Forms.Tests.ClipboardTests.RoundTrip_TypedAndRuntimeDataObject_SupportsTypedInterface
- [ ] System.Windows.Forms.Tests.ClipboardTests.RoundTrip_UntypedDataObject_SupportsTypedInterface
- [ ] Microsoft.VisualBasic.Forms.Tests.WindowsFormsApplicationBaseTests.ShowHideSplashScreenSuccess

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.