JetBrains-Research / JetBrains-Research/TestSpark

[Prompt-scarcity] List of cases when the prompt is not sufficient (feel free to add)

Open
#307 0 comments 0 reactions 0 assignees View on GitHub
new feature
Dominant language
Kotlin
Stars
88
Forks
24
PR merge metrics
No merged PRs in 30d

Description

### 1. Not adding enum properties to the prompt when the enum is the part of function parameters

We want to test the following class inside the TestSpark

```
org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy
```

the prompt looks like this

--------------------------------------------------
Generate unit tests in kotlin for org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy to achieve 100% line coverage for this class.
Dont use @Before and @After test methods.
Make tests as atomic as possible.
All tests should be for JUnit 5.
In case of mocking, use Mockito 5. But, do not use mocking for all tests.
Name all methods according to the template - [MethodUnderTest][Scenario]Test, and use only English letters.
The source code of class under test is as follows:
```
package org.jetbrains.research.testspark.core.test.strategies

import org.jetbrains.research.testspark.core.data.JUnitVersion
import org.jetbrains.research.testspark.core.test.TestCaseParseResult
import org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy
import org.jetbrains.research.testspark.core.test.data.TestCaseGeneratedByLLM
import org.jetbrains.research.testspark.core.test.data.TestLine
import org.jetbrains.research.testspark.core.test.data.TestLineType
import org.jetbrains.research.testspark.core.test.data.TestSuiteGeneratedByLLM
class JUnitTestSuiteParserStrategy : TestSuiteParserStrategy {

}
```
org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy extends org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy. The source code of org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy is:
```
package org.jetbrains.research.testspark.core.test

import org.jetbrains.research.testspark.core.data.JUnitVersion
import org.jetbrains.research.testspark.core.test.data.TestSuiteGeneratedByLLM
interface TestSuiteParserStrategy {
fun parseTestSuite(
rawText: String,
junitVersion: JUnitVersion,
importPattern: Regex,
packageName: String,
testNamePattern: String,
): TestSuiteGeneratedByLLM?
}
```
Generate public classes and public methods. Response only a code with tests between `, do not provide any other text.

--------------------------------------------------

As you can notice, in this prompt we do not have any information about parameters in `JUnitVersion`, therefore generated tests will not be compilable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.