Shopify / Shopify/roast

Generalize VCR testing infrastructure for user workflows

Open
#835 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Ruby
Stars
1.2k
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Context

The VCR testing infrastructure was successfully implemented for the examples tests in PR #655. Currently, it lives entirely within Roast's internal test suite (test/examples/support/functional_test.rb under the Examples::FunctionalTest namespace) and is only usable for testing the built-in examples.

Sam's ask: "now that you have the VCR tests merged, I think another high impact item related to that would be to generalize the system so anybody can use it to write tests for their roast workflows, not just our examples."

A planning doc already exists at docs/TESTING_GENERALIZATION_PLAN.md with detailed implementation steps.

Goal

Make it easy for anyone to write reliable, fast tests for their Roast workflows without needing real API credentials or incurring costs.

What Needs to Change

Current State
  • Examples::FunctionalTest is a private test base class in test/examples/support/
  • VCR config is hardcoded in test/test_helper.rb (cassette dir, fake credentials, base URL)
  • in_sandbox copies examples/ directory — not useful for external workflows
  • VCR and WebMock are dev-only dependencies (require: false)
  • No lib/roast/testing/ exists — zero public test helpers
  • No documentation on testing workflows
Target State
  • Public Roast::Testing::WorkflowTest base class users can inherit from
  • Configurable VCR setup (cassette dir, credentials, base URL)
  • in_sandbox works with any workflow directory, not just examples/
  • VCR is optional (graceful degradation when not installed)
  • Clear documentation and annotated example tests

Implementation Plan

Phase 1: Minimal Viable Version
  1. Extract test helper to public API — Move FunctionalTest to lib/roast/testing/workflow_test.rb as Roast::Testing::WorkflowTest, parameterize hardcoded paths
  2. Make VCR optional — Runtime check with defined?(VCR), graceful fallback for users without it
  3. Write testing documentationdocs/TESTING.md with quick start, recording, assertions, troubleshooting
  4. Add annotated example testexamples/test/ showing real test patterns
Phase 2: Polish
  1. Generator commandroast test:init to scaffold test boilerplate (test_helper.rb, example test, cassette dir, .gitignore)

User Experience Target

require "roast/testing/workflow_test"

class MyWorkflowTest < Roast::Testing::WorkflowTest
  test "my workflow produces expected output" do
    stdout, stderr = in_sandbox :my_workflow do
      Roast::Workflow.from_file("path/to/my_workflow.rb", params)
    end
    assert_empty stderr
    assert_includes stdout, "expected output"
  end
end

Recording: RECORD_VCR=true bundle exec ruby test/my_workflow_test.rb
Replay: bundle exec ruby test/my_workflow_test.rb

Open Questions

  • Naming: Roast::Testing::WorkflowTest vs Roast::TestHelper vs Roast::WorkflowTestCase?
  • Should VCR config be provided via a DSL method or class-level config?
  • Should we support RSpec in addition to Minitest?
  • Should roast test:init be part of Phase 1?

Related

  • PR #655: VCR infrastructure for examples (merged)
  • PR #582: Template shorthand syntax (merged)
  • docs/TESTING_GENERALIZATION_PLAN.md: Existing planning document
  • Issue #570: DSL migration and testing improvements

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/TESTING_GENERALIZATION_PLAN.md, then inspect test/examples/support/functional_test.rb and test/test_helper.rb to understand the existing VCR setup and sandbox behavior. Compare those details with the Phase 1 target, including the public helper, optional VCR support, documentation, and annotated example tests; the work is done when users can test arbitrary workflows without real credentials.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
developer-experience, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.