Generalize VCR testing infrastructure for user workflows
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 1.2k
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Context
The VCR testing infrastructure was successfully implemented for the examples tests in PR #655. Currently, it lives entirely within Roast's internal test suite (test/examples/support/functional_test.rb under the Examples::FunctionalTest namespace) and is only usable for testing the built-in examples.
Sam's ask: "now that you have the VCR tests merged, I think another high impact item related to that would be to generalize the system so anybody can use it to write tests for their roast workflows, not just our examples."
A planning doc already exists at docs/TESTING_GENERALIZATION_PLAN.md with detailed implementation steps.
Goal
Make it easy for anyone to write reliable, fast tests for their Roast workflows without needing real API credentials or incurring costs.
What Needs to Change
Current State
Examples::FunctionalTestis a private test base class intest/examples/support/- VCR config is hardcoded in
test/test_helper.rb(cassette dir, fake credentials, base URL) in_sandboxcopiesexamples/directory — not useful for external workflows- VCR and WebMock are dev-only dependencies (
require: false) - No
lib/roast/testing/exists — zero public test helpers - No documentation on testing workflows
Target State
- Public
Roast::Testing::WorkflowTestbase class users can inherit from - Configurable VCR setup (cassette dir, credentials, base URL)
in_sandboxworks with any workflow directory, not justexamples/- VCR is optional (graceful degradation when not installed)
- Clear documentation and annotated example tests
Implementation Plan
Phase 1: Minimal Viable Version
- Extract test helper to public API — Move
FunctionalTesttolib/roast/testing/workflow_test.rbasRoast::Testing::WorkflowTest, parameterize hardcoded paths - Make VCR optional — Runtime check with
defined?(VCR), graceful fallback for users without it - Write testing documentation —
docs/TESTING.mdwith quick start, recording, assertions, troubleshooting - Add annotated example test —
examples/test/showing real test patterns
Phase 2: Polish
- Generator command —
roast test:initto scaffold test boilerplate (test_helper.rb, example test, cassette dir, .gitignore)
User Experience Target
require "roast/testing/workflow_test"
class MyWorkflowTest < Roast::Testing::WorkflowTest
test "my workflow produces expected output" do
stdout, stderr = in_sandbox :my_workflow do
Roast::Workflow.from_file("path/to/my_workflow.rb", params)
end
assert_empty stderr
assert_includes stdout, "expected output"
end
end
Recording: RECORD_VCR=true bundle exec ruby test/my_workflow_test.rb
Replay: bundle exec ruby test/my_workflow_test.rb
Open Questions
- Naming:
Roast::Testing::WorkflowTestvsRoast::TestHelpervsRoast::WorkflowTestCase? - Should VCR config be provided via a DSL method or class-level config?
- Should we support RSpec in addition to Minitest?
- Should
roast test:initbe part of Phase 1?
Related
- PR #655: VCR infrastructure for examples (merged)
- PR #582: Template shorthand syntax (merged)
docs/TESTING_GENERALIZATION_PLAN.md: Existing planning document- Issue #570: DSL migration and testing improvements
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/TESTING_GENERALIZATION_PLAN.md, then inspect test/examples/support/functional_test.rb and test/test_helper.rb to understand the existing VCR setup and sandbox behavior. Compare those details with the Phase 1 target, including the public helper, optional VCR support, documentation, and annotated example tests; the work is done when users can test arbitrary workflows without real credentials.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- developer-experience, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100