approvals / approvals/ApprovalTests.cpp
Catch2 integration does not handle Generators inside sections
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 339
- Forks
- 55
- PR merge metrics
- No merged PRs in 30d
Description
Per Catch2 documentation:
GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope.
[...]
The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION
I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:
// Approvals/integrations/catch/Cathc2Approvals.h
struct Catch2TestCommitRevert : Catch::TestEventListenerBase
{
...
virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
{
currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
}
...
};
In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).
The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:
enum class Keyword
{
Unknown, Foo, Bar, Quux
};
Keyword parseKeyword(const std::string_view word);
SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
const std::string_view word = GENERATE("foo", "bar", "quux");
Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ApprovalTests/integrations/catch/Catch2Approvals.h, especially sectionStarting, then compare its use of Catch::SectionInfo with the Catch2 generators documentation and SectionInfo definition linked in the issue. Investigate how generator executions are exposed during section handling. Done means approvals inside a generated section are uniquely registered for each generator variant rather than being shared across iterations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100