OpenBMB / OpenBMB/ChatDev

CodeReviewModification never shows the programmer the six review regulations, so fixes are made blind to the checklist they are re-reviewed against

Open Beginner friendly
#650 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.3k
Forks
4.3k
PR merge metrics
No merged PRs in 30d

Description

Branch: chatdev1.0 (Default company config). Line numbers below refer to that branch; the same prompts exist on other configs derived from Default.

Summary

In CompanyConfig/Default/PhaseConfig.json, the CodeReviewComment phase prompt (lines 134–153) gives the Code Reviewer six regulations (lines 146–151: imports, implemented methods, comments, bugs, task conformance, interaction logic) and instructs them to "check the above regulations one by one" — but then to "propose one comment with the highest priority" (line 152).

The CodeReviewModification phase prompt (lines 155–178) passes the Programmer only:

"Comments on Codes:",
"\"{comments}\"",

(lines 166–167). The six-regulation checklist itself never reaches the modification prompt. So in each review cycle:

  • the reviewer knows all six regulations but reports only one violation;
  • the programmer fixes that one violation without knowing the other five criteria the next CodeReviewComment pass will judge the code against.

Triggering condition

ChatChainConfig.json caps the CodeReview ComposedPhase at cycleNum: 3 (lines 31–48), so at most three single comments are ever applied per run. Whenever the code has more than three regulation violations — or a fix regresses a different regulation (e.g. adding a method without importing what it needs) — known violations survive the entire review loop and ship in the final software. Unimplemented methods and missing imports, the exact things regulations 1) and 2) exist to catch, are the common survivors.

Suggested fix

Pure prompt change to CompanyConfig/Default/PhaseConfig.json: include the same six regulations in the CodeReviewModification phase prompt (before or after "Comments on Codes:"), and ask the Programmer to keep all regulations satisfied while addressing the comment. This lets each modification avoid regressing the criteria it will immediately be re-reviewed against, making the three cycles count. No code changes needed.

Found during an academic study that writes behavioral specifications of multi-agent systems and statically analyzes them; the reviewer-to-programmer information loss surfaced when specifying what each phase's prompt actually carries between roles.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in CompanyConfig/Default/PhaseConfig.json with the CodeReviewComment and CodeReviewModification prompts, then check ChatChainConfig.json to understand the three-cycle review limit. Update the modification prompt so it includes the six regulations and asks the Programmer to preserve them while addressing the comment. Done means the Programmer receives the same checklist used for re-review; inspect derived configurations for the same prompt pattern.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.