ASSERT-KTH / ASSERT-KTH/human-eval-java
Incorrect Testing Failures Message
- Dominant language
- Java
- Stars
- 22
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
I've observed that the test reports for some bugs seem to get the “expected“ value and “but was” value obtained reversed.
For example:
Bug_ID: `ADD`
Buggy_Code:
```java
/* Add two numbers x and y
>>> add(2, 3)
5
>>> add(5, 7)
12 */
public class ADD {
public static int add(int x, int y) {
return x | y;
}
}
```
Test Cases:
```java
public class TEST_ADD {
@org.junit.Test(timeout = 3000)
public void test_0() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(0, 1);
org.junit.Assert.assertEquals(
result, 1
);
}
@org.junit.Test(timeout = 3000)
public void test_1() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(1, 0);
org.junit.Assert.assertEquals(
result, 1
);
}
@org.junit.Test(timeout = 3000)
public void test_2() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(2, 3);
org.junit.Assert.assertEquals(
result, 5
);
}
@org.junit.Test(timeout = 3000)
public void test_3() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(5, 7);
org.junit.Assert.assertEquals(
result, 12
);
}
@org.junit.Test(timeout = 3000)
public void test_4() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(7, 5);
org.junit.Assert.assertEquals(
result, 12
);
}
}
```
Current (Incorrect) Testing Failures Message:
```
Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<5>
at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)
test_3(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)
test_4(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)
```
Correct Testing Failures Message:
```
Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<3>
at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)
test_3(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)
test_4(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.