mozilla / mozilla/bugbot

CC test authors / triage owners on Phabricator patches that disable a test

Open
#1,127 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area-Intermittents
Dominant language
Python
Stars
68
Forks
92
Avg merge
2d 2h
Merged PRs (30d)
16

Description

If a test fails too frequently and sheriffs did not have any luck getting traction to fix it, they will simply skip it. We try to let the triage-owner of a component know via needinfo, but this is:

  1. Inconsistent
  2. Not actually a needinfo (just an fyi)
  3. Usually results in no action
  4. Probably annoying to triage-owners of large components with lots of tests

It would be nice if we could simply CC the test author (if we can find them) and the triage owner (because why not) right on the patch in Phabricator. At this point, they have already been warned that the test will be disabled (see also issue #1126), so they should be CC'ed rather than requested for review.

We'll need to determine which patches qualify as "disabling a test". Some basic criteria might be:

  1. Submitted by a sheriff
  2. Only touches a test manifest (list of test manifests can be extracted from moz.build metadata)

Some text processing would also likely be needed to determine which test in the manifest is being modified.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing bugbot's Phabricator patch handling and the moz.build metadata used to identify test manifests; issue #1126 provides related context. Determine how to recognize patches that disable a test, find the affected test and its author, and CC that author and the component triage owner without requesting review.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
release, testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.