citrusframework / citrusframework/citrus
DataProvider gets invoked many times
- Dominant language
- Java
- Stars
- 485
- Forks
- 155
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 7
Description
**Description**
Assume you have a TestNG DataProvider that returns n parameter sets.
If you execute a parameterized Citrus test / DataProvider, the DataProvider method gets invoked n+1 times.
**Citrus Version**
2.7.7
**Expected behavior**
The DataProvider method should be invoked exactly *once* at the begin of the test executions.
**Actual behavior**
The DataProvider method gets invoked once at the begin of the test executions and also gets invoked for every of the n parameter sets (resulting in n+1 executions).
**Test case sample**
```
public class TestIT extends TestNGCitrusTestRunner {
@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}
@Parameters({"dataProviderValue"})
@CitrusTest
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// do nothing
}
}
```
results in the output:
```
14:19:38,379 DEBUG citrus.Citrus| Loading Citrus application properties
14:19:38,401 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,902 INFO port.LoggingReporter| .__ __
14:19:38,902 INFO port.LoggingReporter| ____ |__|/ |________ __ __ ______
14:19:38,902 INFO port.LoggingReporter| _/ ___\| \ __\_ __ \ | \/ ___/
14:19:38,902 INFO port.LoggingReporter| \ \___| || | | | \/ | /\___ \
14:19:38,902 INFO port.LoggingReporter| \___ >__||__| |__| |____//____ >
14:19:38,902 INFO port.LoggingReporter| \/ \/
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| C I T R U S T E S T S 2.7.7
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 DEBUG port.LoggingReporter| BEFORE TEST SUITE
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
14:19:38,903 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 INFO port.LoggingReporter|
CALLING DATAPROVIDER
14:19:38,919 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,926 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 DEBUG citrus.TestCase| Initializing test case
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:38,927 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:38,927 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:19:38,927 DEBUG citrus.TestCase| Test variables:
14:19:38,927 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:38,927 DEBUG citrus.TestCase| dataProviderValue = a
14:19:38,927 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:39,929 INFO port.LoggingReporter|
14:19:39,930 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:39,930 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,930 INFO port.LoggingReporter|
14:19:39,931 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,932 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 DEBUG citrus.TestCase| Initializing test case
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:39,932 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:39,932 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'b'
14:19:39,932 DEBUG citrus.TestCase| Test variables:
14:19:39,932 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:39,932 DEBUG citrus.TestCase| dataProviderValue = b
14:19:39,932 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:40,932 INFO port.LoggingReporter|
14:19:40,933 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:40,933 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,933 INFO port.LoggingReporter|
14:19:40,934 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,935 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 DEBUG citrus.TestCase| Initializing test case
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:40,935 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:40,936 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'c'
14:19:40,936 DEBUG citrus.TestCase| Test variables:
14:19:40,936 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:40,936 DEBUG citrus.TestCase| dataProviderValue = c
14:19:40,936 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:41,936 INFO port.LoggingReporter|
14:19:41,936 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:41,936 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,936 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 DEBUG port.LoggingReporter| AFTER TEST SUITE
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| AFTER TEST SUITE: SUCCESS
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| CITRUS TEST RESULTS
14:19:41,940 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TOTAL: 3
14:19:41,942 DEBUG port.LoggingReporter| SKIPPED: 0 (0.0%)
14:19:41,942 INFO port.LoggingReporter| FAILED: 0 (0.0%)
14:19:41,943 INFO port.LoggingReporter| SUCCESS: 3 (100.0%)
14:19:41,943 INFO port.LoggingReporter|
14:19:41,943 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,943 DEBUG report.HtmlReporter| Generating HTML test report
14:19:41,944 DEBUG util.FileUtils| Reading file resource: 'test-detail.html' (encoding is 'UTF-8')
14:19:41,961 DEBUG util.FileUtils| Reading file resource: 'test-report.html' (encoding is 'UTF-8')
14:19:41,962 INFO ctOutputFileReporter| Generated test report: target/citrus-reports/citrus-test-results.html
14:19:41,963 DEBUG report.JUnitReporter| Generating JUnit test report
14:19:41,964 DEBUG util.FileUtils| Reading file resource: 'junit-test.xml' (encoding is 'UTF-8')
14:19:41,969 DEBUG util.FileUtils| Reading file resource: 'junit-report.xml' (encoding is 'UTF-8')
===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
Process finished with exit code 0
```
As you can see the line
```
CALLING DATAPROVIDER
```
is present in the output 4 times, while it should be present only once.
**Note**
To make sure that this is actually a Citrus issue I created the same test without Citrus / only with TestNG - without Citrus the DataProvider method gets invoked only *once* (expected behavior).
```
public class TestWithoutCitrusIT {
@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}
@Parameters({"dataProviderValue"})
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// print value to make sure that the test gets invoked n times
System.out.println("DataProviderValue " + dataProviderValue);
}
}
```
results in the output
```
CALLING DATAPROVIDER
DataProviderValue a
DataProviderValue b
DataProviderValue c
===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
Process finished with exit code 0
```
As you can see the line
```
CALLING DATAPROVIDER
```
is present in the output only *once*, which is the expected behavior.
Contributor guide
Research direction
Start with the TestNGCitrusTestRunner integration and the DataProvider handling around the sample TestIT.testIT method. Compare its execution path with the TestNG-only example and trace why dataProvider() is called once initially and again for each parameter set. Add a regression test that verifies one invocation while retaining three test executions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100