microsoft / microsoft/BotFramework-WebChat

[Test reliability] Eliminate false positive and reduce wait time on verifying speech recognition is not restarted

Open
#4,046 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request technical-debt
Dominant language
HTML
Stars
1.8k
Forks
1.6k
Avg merge
22h 58m
Merged PRs (30d)
2

Description

Feature Request

tl;dr we are waiting for "that signal should NOT be fired". This can introduce false positives and unnecessary wait time for positives.

There is a test reliability issue for turning on microphone after input hints:

  • After receiving input hint of expecting, we will turn on microphone
  • We use Redux + render() to turn on microphone

Combining both in a busy environment, the microphone may not turn on immediately. We need to wait for the condition, more-or-less:

became(() => speechRecognitionStartCalled(), ...)

As we also test for negative case: after receiving input hint of ignoring, we should NOT turn on the microphone. But unfortunately, we don't have any signal to detect if Web Chat decided NOT to turn on microphone.

So, we are waiting for X seconds to make sure the speech recognition is not restarted. This is not great for a few reasons

  • False positives: X seconds may not be long enough to make sure speech recognition is not started
  • In positive case, we lengthened the tests

To fix this bug, add/find a good signal: after receiving input hint of ignoring, wait for a signal from Web Chat that it decided not to fire the speech recognition signal.

The following is a code snippet we were updating our test. This needs to further updated to make sure we have a good signal to make the last line faster.

  test('should not turn on microphone if initiated via typing', async () => {
    const { driver, pageObjects } = await setupWebDriver({
      props: {
        webSpeechPonyfillFactory: () => window.WebSpeechMock
      }
    });

    await driver.wait(uiConnected(), timeouts.directLine);

    await pageObjects.sendMessageViaMicrophone('hint expecting');

    await driver.wait(minNumActivitiesShown(2), timeouts.directLine);

-   await expect(speechRecognitionStartCalled().fn(driver)).resolves.toBeFalsy();
+   await expect(() => driver.wait(speechRecognitionStartCalled(), timeouts.ui)).rejects.toThrow('Waiting SpeechRecognition.start to be called');
  });

[feature-request]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test named "should not turn on microphone if initiated via typing" and inspect setupWebDriver, speechRecognitionStartCalled, and the input-hint flow. Determine which Web Chat signal can confirm that speech recognition will not restart, then update the test to wait on that signal instead of a fixed delay. Done means the negative case avoids false positives and unnecessary waiting.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
frontend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.