microsoft / microsoft/BotFramework-WebChat

[Test reliability] Bucket of tests need reliability improvements

Open
#4,080 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request technical-debt
Dominant language
HTML
Stars
1.8k
Forks
1.6k
Avg merge
22h 58m
Merged PRs (30d)
2

Description

Feature Request

accessibility.adaptiveCard.disabled.focus.withShowCard.accessibility requirement / disabling Adaptive Card with "Action.ShowCard"

image

accessibility requirement attachments in live region audio
Fixed

File: __tests__/html/accessibility.liveRegionAttachment.audio.js

image

accessibility.liveRegionActivity.performance.accessibility requirement / should not run renderMarkdown in excess

Sometimes, we render 140 times. And sometimes 130 times.

That extra 10 renderings come at the very last moment just before Web Chat is unmounted. We need to investigate what is causing that extra 10 renderings.

We were expecting 140 vs. 120. I am unsure if it should be 130 vs. 120 instead.

Error: expect(received).toBe(expected) // Object.is equality

Expected: 20
Received: 10
    at http://webchat2/__tests__/html/accessibility.liveRegionActivity.performance:86:3
Working console log (140 vs. 120 renderings) 
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 1
webchat-es5.js:sourcemap:262  {text: '1'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 1
accessibility.liveRegionActivity.performance:43 1 live
2accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 2
webchat-es5.js:sourcemap:262  {text: '2'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 2
accessibility.liveRegionActivity.performance:43 2 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 3
webchat-es5.js:sourcemap:262  {text: '3'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 3
accessibility.liveRegionActivity.performance:43 3 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 4
webchat-es5.js:sourcemap:262  {text: '4'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 4
accessibility.liveRegionActivity.performance:43 4 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 5
webchat-es5.js:sourcemap:262  {text: '5'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 5
accessibility.liveRegionActivity.performance:43 5 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 6
webchat-es5.js:sourcemap:262  {text: '6'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 6
accessibility.liveRegionActivity.performance:43 6 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 7
webchat-es5.js:sourcemap:262  {text: '7'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 7
accessibility.liveRegionActivity.performance:43 7 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 8
webchat-es5.js:sourcemap:262  {text: '8'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 8
accessibility.liveRegionActivity.performance:43 8 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 9
webchat-es5.js:sourcemap:262  {text: '9'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 9
accessibility.liveRegionActivity.performance:43 9 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 10
accessibility.liveRegionActivity.performance:43 10
webchat-es5.js:sourcemap:262  {text: '10'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 10
accessibility.liveRegionActivity.performance:43 10 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 10
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 10
accessibility.liveRegionActivity.performance:61 ---------- UNMOUNTED -----------
accessibility.liveRegionActivity.performance:89 {numRenderMarkdownWithLiveRegionActivity: 140, numRenderMarkdownWithoutLiveRegionActivity: 120}
VM22503:4 ✔️ DONE
Not working (130 vs. 120 renderings) 
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 1
webchat-es5.js:sourcemap:262  {text: '1'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 1
accessibility.liveRegionActivity.performance:43 1 live
2accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 2
webchat-es5.js:sourcemap:262  {text: '2'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 2
accessibility.liveRegionActivity.performance:43 2 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 3
webchat-es5.js:sourcemap:262  {text: '3'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 3
accessibility.liveRegionActivity.performance:43 3 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 4
webchat-es5.js:sourcemap:262  {text: '4'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 4
accessibility.liveRegionActivity.performance:43 4 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 5
webchat-es5.js:sourcemap:262  {text: '5'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 5
accessibility.liveRegionActivity.performance:43 5 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 6
webchat-es5.js:sourcemap:262  {text: '6'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 6
accessibility.liveRegionActivity.performance:43 6 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 7
webchat-es5.js:sourcemap:262  {text: '7'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 7
accessibility.liveRegionActivity.performance:43 7 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 8
webchat-es5.js:sourcemap:262  {text: '8'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 8
accessibility.liveRegionActivity.performance:43 8 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 9
webchat-es5.js:sourcemap:262  {text: '9'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 9
accessibility.liveRegionActivity.performance:43 9 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 10
accessibility.liveRegionActivity.performance:43 10
webchat-es5.js:sourcemap:262  {text: '10'}
webchat-es5.js:sourcemap:262 renderMarkdownAsHTML 10
accessibility.liveRegionActivity.performance:43 10 live
accessibility.liveRegionActivity.performance:43 1
accessibility.liveRegionActivity.performance:43 2
accessibility.liveRegionActivity.performance:43 3
accessibility.liveRegionActivity.performance:43 4
accessibility.liveRegionActivity.performance:43 5
accessibility.liveRegionActivity.performance:43 6
accessibility.liveRegionActivity.performance:43 7
accessibility.liveRegionActivity.performance:43 8
accessibility.liveRegionActivity.performance:43 9
accessibility.liveRegionActivity.performance:43 10
accessibility.liveRegionActivity.performance:61 ---------- UNMOUNTED -----------
accessibility.liveRegionActivity.performance:89 {numRenderMarkdownWithLiveRegionActivity: 130, numRenderMarkdownWithoutLiveRegionActivity: 120}
VM21913:8 Error: expect(received).toBe(expected) // Object.is equality

Expected: 20

Received: 10

at accessibility.liveRegionActivity.performance:94:31

eval @ VM21913:8

eval @ VM21913:30

executeScript @ VM21912:478

(anonymous) @ VM21912:483

callFunction @ VM21912:446

(anonymous) @ VM21912:460

(anonymous) @ VM21912:461

VM21913:10 ❌ FAILED

[feature-request]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/html/accessibility.liveRegionAttachment.audio.js and the accessibility.liveRegionActivity.performance test. Reproduce the reported render-count variation and inspect the Adaptive Card disabled-focus requirement. Done means the affected tests reliably pass and their accessibility expectations are clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, javascript
Domain
accessibility, testing, web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.