Model answers the same on a classification task
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I use the Vicuna-7B model helping me with the clone detection task, but the model just output the same answers with **ONE-SHOT**. Here's one of my prompts:
A chat between a curious user and an artificial intelligence assistant.
### User: Is there a colne relation between the Code1 and Code2, and respond to YES or NO.
Code1: private void setNodekeyInJsonResponse(String service) throws Exception { String filename = this.baseDirectory + service + \".json\"; Scanner s = new Scanner(new File(filename)); PrintWriter fw = new PrintWriter(new File(filename + \".new\")); while (s.hasNextLine()) { fw.println(s.nextLine().replaceAll(\"NODEKEY\", this.key)); } s.close(); fw.close(); (new File(filename + \".new\")).renameTo(new File(filename)); }.
Code2: public void transform(String style, String spec, OutputStream out) throws IOException { URL url = new URL(rootURL, spec); InputStream in = new PatchXMLSymbolsStream(new StripDoctypeStream(url.openStream())); transform(style, in, out); in.close(); }.
### Assistant: YES.
### User: Is there a colne relation between the Code1 and Code2, and respond to YES or NO.
Code1: private void loadProperties() { if (properties == null) { properties = new Properties(); try { URL url = getClass().getResource(propsFile); properties.load(url.openStream()); } catch (IOException ioe) { ioe.printStackTrace(); } } }.
Code2: private void update(String statement, SyrupConnection con, boolean do_log) throws Exception { Statement s = null; try { s = con.createStatement(); s.executeUpdate(statement); con.commit(); } catch (Throwable e) { if (do_log) { logger.log(Level.INFO, \"Update failed. Transaction is rolled back\", e); } con.rollback(); } }.
### Assistant:
I tried many different code pairs and shots, but the model just keep answering **NO** even if there's a clone relation between two code snippets. Do you have any ideas about it?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is named. Start by reproducing the reported one-shot Vicuna-7B prompt with code pairs that should and should not be clones, then determine whether the behavior is a prompting or model limitation; done means identifying a reproducible actionable defect or documenting that no repository change is specified.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100