citrusframework / citrusframework/citrus

DataProvider gets invoked many times

Open
#502 6 comments 0 reactions 0 assignees View on GitHub
enhancement prio-low
Dominant language
Java
Stars
485
Forks
155
Avg merge
2d 10h
Merged PRs (30d)
7

Description

**Description**
Assume you have a TestNG DataProvider that returns n parameter sets.

If you execute a parameterized Citrus test / DataProvider, the DataProvider method gets invoked n+1 times.

**Citrus Version**
2.7.7

**Expected behavior**
The DataProvider method should be invoked exactly *once* at the begin of the test executions.

**Actual behavior**
The DataProvider method gets invoked once at the begin of the test executions and also gets invoked for every of the n parameter sets (resulting in n+1 executions).

**Test case sample**
```
public class TestIT extends TestNGCitrusTestRunner {

@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}

@Parameters({"dataProviderValue"})
@CitrusTest
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// do nothing
}
}
```

results in the output:
```
14:19:38,379 DEBUG citrus.Citrus| Loading Citrus application properties
14:19:38,401 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,902 INFO port.LoggingReporter| .__ __
14:19:38,902 INFO port.LoggingReporter| ____ |__|/ |________ __ __ ______
14:19:38,902 INFO port.LoggingReporter| _/ ___\| \ __\_ __ \ | \/ ___/
14:19:38,902 INFO port.LoggingReporter| \ \___| || | | | \/ | /\___ \
14:19:38,902 INFO port.LoggingReporter| \___ >__||__| |__| |____//____ >
14:19:38,902 INFO port.LoggingReporter| \/ \/
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| C I T R U S T E S T S 2.7.7
14:19:38,902 INFO port.LoggingReporter|
14:19:38,902 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 DEBUG port.LoggingReporter| BEFORE TEST SUITE
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter|
14:19:38,903 INFO port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
14:19:38,903 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,903 INFO port.LoggingReporter|
CALLING DATAPROVIDER
14:19:38,919 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:38,926 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:38,926 INFO port.LoggingReporter|
14:19:38,926 DEBUG citrus.TestCase| Initializing test case
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:38,927 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:38,927 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:38,927 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'a'
14:19:38,927 DEBUG citrus.TestCase| Test variables:
14:19:38,927 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:38,927 DEBUG citrus.TestCase| dataProviderValue = a
14:19:38,927 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:39,929 INFO port.LoggingReporter|
14:19:39,930 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:39,930 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,930 INFO port.LoggingReporter|
14:19:39,931 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:39,932 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:39,932 INFO port.LoggingReporter|
14:19:39,932 DEBUG citrus.TestCase| Initializing test case
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:39,932 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:39,932 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:39,932 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'b'
14:19:39,932 DEBUG citrus.TestCase| Test variables:
14:19:39,932 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:39,932 DEBUG citrus.TestCase| dataProviderValue = b
14:19:39,932 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:40,932 INFO port.LoggingReporter|
14:19:40,933 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:40,933 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,933 INFO port.LoggingReporter|
14:19:40,934 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
CALLING DATAPROVIDER
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:40,935 DEBUG port.LoggingReporter| STARTING TEST TestIT.testIT
14:19:40,935 INFO port.LoggingReporter|
14:19:40,935 DEBUG citrus.TestCase| Initializing test case
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'TestIT.testIT'
14:19:40,935 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'test.citrus.DataProviderParallel'
14:19:40,935 DEBUG citrus.TestCase| Initializing test parameter 'dataProviderValue' as variable
14:19:40,936 DEBUG context.TestContext| Setting variable: dataProviderValue with value: 'c'
14:19:40,936 DEBUG citrus.TestCase| Test variables:
14:19:40,936 DEBUG citrus.TestCase| citrus.test.name = TestIT.testIT
14:19:40,936 DEBUG citrus.TestCase| dataProviderValue = c
14:19:40,936 DEBUG citrus.TestCase| citrus.test.package = test.citrus.DataProviderParallel
14:19:41,936 INFO port.LoggingReporter|
14:19:41,936 INFO port.LoggingReporter| TEST SUCCESS TestIT.testIT (test.citrus.DataProviderParallel)
14:19:41,936 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,936 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 DEBUG port.LoggingReporter| AFTER TEST SUITE
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| AFTER TEST SUITE: SUCCESS
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,940 INFO port.LoggingReporter|
14:19:41,940 INFO port.LoggingReporter| CITRUS TEST RESULTS
14:19:41,940 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter| TestIT.testIT .................................................. SUCCESS
14:19:41,942 INFO port.LoggingReporter|
14:19:41,942 INFO port.LoggingReporter| TOTAL: 3
14:19:41,942 DEBUG port.LoggingReporter| SKIPPED: 0 (0.0%)
14:19:41,942 INFO port.LoggingReporter| FAILED: 0 (0.0%)
14:19:41,943 INFO port.LoggingReporter| SUCCESS: 3 (100.0%)
14:19:41,943 INFO port.LoggingReporter|
14:19:41,943 INFO port.LoggingReporter| ------------------------------------------------------------------------
14:19:41,943 DEBUG report.HtmlReporter| Generating HTML test report
14:19:41,944 DEBUG util.FileUtils| Reading file resource: 'test-detail.html' (encoding is 'UTF-8')
14:19:41,961 DEBUG util.FileUtils| Reading file resource: 'test-report.html' (encoding is 'UTF-8')
14:19:41,962 INFO ctOutputFileReporter| Generated test report: target/citrus-reports/citrus-test-results.html
14:19:41,963 DEBUG report.JUnitReporter| Generating JUnit test report
14:19:41,964 DEBUG util.FileUtils| Reading file resource: 'junit-test.xml' (encoding is 'UTF-8')
14:19:41,969 DEBUG util.FileUtils| Reading file resource: 'junit-report.xml' (encoding is 'UTF-8')

===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

Process finished with exit code 0
```

As you can see the line
```
CALLING DATAPROVIDER
```
is present in the output 4 times, while it should be present only once.

**Note**
To make sure that this is actually a Citrus issue I created the same test without Citrus / only with TestNG - without Citrus the DataProvider method gets invoked only *once* (expected behavior).

```
public class TestWithoutCitrusIT {

@DataProvider(name = "dp")
public static Object[][] dataProvider() {
System.out.println("CALLING DATAPROVIDER");
return Stream.of("a", "b", "c").map(s -> new Object[]{s}).toArray(Object[][]::new);
}

@Parameters({"dataProviderValue"})
@Test(dataProvider = "dp")
public void testIT(String dataProviderValue) {
// print value to make sure that the test gets invoked n times
System.out.println("DataProviderValue " + dataProviderValue);
}
}
```

results in the output
```
CALLING DATAPROVIDER
DataProviderValue a
DataProviderValue b
DataProviderValue c

===============================================
Default Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

Process finished with exit code 0
```

As you can see the line
```
CALLING DATAPROVIDER
```
is present in the output only *once*, which is the expected behavior.

Contributor guide

Open the contributing guide

Research direction

Start with the TestNGCitrusTestRunner integration and the DataProvider handling around the sample TestIT.testIT method. Compare its execution path with the TestNG-only example and trace why dataProvider() is called once initially and again for each parameter set. Add a regression test that verifies one invocation while retaining three test executions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.