approvals / approvals/ApprovalTests.cpp

Catch2 integration does not handle Generators inside sections

Open
#205 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
339
Forks
55
PR merge metrics
No merged PRs in 30d

Description

Per Catch2 documentation:

GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope.
[...]
The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION

I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:

// Approvals/integrations/catch/Cathc2Approvals.h

struct Catch2TestCommitRevert : Catch::TestEventListenerBase
{
    ...
    virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
    {
        currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
    }
    ...
};

In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).

The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:

enum class Keyword
{
    Unknown, Foo, Bar, Quux
};

Keyword parseKeyword(const std::string_view word);

SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
     const std::string_view word = GENERATE("foo", "bar", "quux");
     Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ApprovalTests/integrations/catch/Catch2Approvals.h, especially sectionStarting, then compare its use of Catch::SectionInfo with the Catch2 generators documentation and SectionInfo definition linked in the issue. Investigate how generator executions are exposed during section handling. Done means approvals inside a generated section are uniquely registered for each generator variant rather than being shared across iterations.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.