luckyframework / luckyframework/lucky_flow

Issue with Flapping Tests

Open
#116 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Crystal
Stars
55
Forks
9
PR merge metrics
No merged PRs in 30d

Description

I'm currently leveraging LuckyFlow to test the entirety of https://luckycasts.com

I've never had an issue in my development environment, but in GitHub Actions I continue to get random tests that will fail with content like this:

Screen Shot 2021-01-20 at 9 25 08 AM

If I re-run the job, everything is fine and the test passes without any changes.

Here's that spec code:

  it "allows admin users to delete topics" do
    flow = TopicManagementFlow.new
    topic = TopicBox.create
    user = UserBox.create &.is_admin(true)

    flow.visit Admin::Topics::Edit.with(topic.id), as: user
    flow.click("@delete-topic-button")
    flow.accept_alert

    flow.should have_current_path(Admin::Topics::Index)
    flow.el("body", text: topic.label).should_not be_on_page
  end

Here's the relevant GitHub workflow definition parts:

name: Lucky App CI

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  RunSpecs:
    runs-on: ubuntu-latest
    container:
      image: crystallang/crystal:0.35.1
    env:
      LUCKY_ENV: test
      DB_HOST: postgres

    services:
      postgres:
        image: postgres:12-alpine
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v2

      - name: Install PostgreSQL client
        run: |
          apt-get update
          apt-get -yqq install libpq-dev postgresql-client

      - name: Install browser
        run: apt-get -yqq install chromium-browser

      - uses: actions/setup-node@v2.1.4
        with:
          node-version: "12.x"
      - name: "Install yarn"
        run: npm install -g yarn

      - name: Get yarn cache directory path
        id: yarn-cache-dir-path
        run: echo "::set-output name=dir::$(yarn cache dir)"

      - name: Set up Yarn cache
        uses: actions/cache@v2
        id: yarn-cache
        with:
          path: |
            ${{ steps.yarn-cache-dir-path.outputs.dir }}
            **/node_modules

          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-

      - name: Set up Crystal cache
        uses: actions/cache@v2
        id: crystal-cache
        with:
          path: |
            ~/.cache/crystal
            bin/ameba
            lib
          key: ${{ runner.os }}-crystal-${{ hashFiles('**/shard.lock') }}
          restore-keys: |
            ${{ runner.os }}-crystal-

      - name: Install shards
        if: steps.crystal-cache.outputs.cache-hit != 'true'
        run: shards check || shards install

      - name: Install yarn packages
        if: steps.yarn-cache.outputs.cache-hit != 'true'
        run: yarn install --frozen-lockfile --no-progress

      - name: Compiling assets
        run: yarn prod

      - name: Run tests
        run: crystal spec

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided TopicManagementFlow spec and the GitHub Actions workflow, then reproduce the intermittent failure by running crystal spec in the stated container and browser setup. Compare failing and successful runs to isolate the source of the flakiness; done means the cause is understood and the workflow passes reliably without reruns.

Written by the indexing model from the issue text.

Assessment

Tech stack
crystal, github-actions
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.