TheAgentCompany / TheAgentCompany/experiments
Analysis of impacts due to different LLM-as-Colleagues
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 24
- Forks
- 15
- PR merge metrics
- No merged PRs in 30d
Description
In the leaderboard, we stick to Claude-Sonnet-3.7 as environment LLM (LLM-as-Colleague and LLM-as-Evaluator).
In this post, we analyze the impact of using different environment LLMs. We compare all 41 tasks that involve conversations with NPCs (colleagues) backed by LLMs.
All experiments used the same agent version and same agent LLM version. Full results are available at:
- Claude 3.7 Sonnet: https://github.com/TheAgentCompany/experiments/tree/main/evaluation/1.0.0/20250510_OpenHands-0.28.1-sonnet-20250219
- GPT-4o: https://github.com/TheAgentCompany/experiments/pull/6
- DeepSeek v3: https://github.com/TheAgentCompany/experiments/pull/7
Caveat: GPT-4o and DeepSeek v3 experiments were run on 07/29/2025, while Claude 3.7 Sonnet experiment was run on 5/10/2025
| Task Name | Claude Sonnet 3.7 Score | GPT-4o Score | DeepSeek v3 Score |
|---|---|---|---|
| admin-ask-for-meeting-feedback | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| admin-ask-for-upgrade-reimbursement | 0.12 | 1.00 ⭐ | 1.00 ⭐ |
| admin-employee-info-reconciliation | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| hr-check-for-invalid-passwords-and-ask-for-valid-passwords | 1.00 ⭐ | 1.00 ⭐ | 0.25 |
| hr-collect-feedbacks | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| hr-collect-multiple-valid-passwords | 0.25 | 1.00 ⭐ | 1.00 ⭐ |
| hr-delete-and-insert-user | 1.00 ⭐ | 1.00 ⭐ | 0.33 |
| hr-get-valid-password | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| hr-new-grad-job-description-2 | 0.12 | 1.00 ⭐ | 0.38 |
| hr-new-grad-job-description-3 | 0.3 | 1.00 ⭐ | 1.00 ⭐ |
| hr-pick-interviewer-1 | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| pm-ask-issue-assignee-for-issue-status-and-update-in-plane | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| pm-check-backlog-update-issues | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| pm-prepare-meeting-with-customers | 0.42 | 1.00 ⭐ | 0.42 |
| pm-schedule-meeting-1 | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| pm-send-notification-to-corresponding-user | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| qa-update-issue-status-according-to-colleagues | 1.00 ⭐ | 1.00 ⭐ | 1.00 ⭐ |
| sde-reply-community-issue-by-asking-npc | 1.00 ⭐ | 1.00 ⭐ | 0.3 |
| admin-get-best-vendor-quote | 0.33 | 0.33 | 0.25 |
| hr-green-card-consultation | 0.33 | 0.33 | 0.33 |
| hr-pick-interviewer-2 | 0.33 | 0.33 | 0.33 |
| sde-create-new-repo | 1.00 ⭐ | 0.33 | 1.00 ⭐ |
| example | 0.3 | 0.3 | 0.3 |
| finance-find-signatories | 0.3 | 0.3 | 0.3 |
| admin-check-employees-budget-and-reply-2 | 0.25 | 0.25 | 0.25 |
| admin-check-employees-budget-and-reply | 0.25 | 0.25 | 0.12 |
| hr-check-attendance-multiple-days-department-with-chat | 0.25 | 0.25 | 0.38 |
| hr-resume-screening | 0.25 | 0.25 | 0.25 |
| pm-plan-personnel-for-new-project | 0.21 | 0.21 | 0.21 |
| pm-ask-for-issue-and-create-in-gitlab | 1.00 ⭐ | 0.2 | 1.00 ⭐ |
| pm-schedule-meeting-2 | 1.00 ⭐ | 0.2 | 1.00 ⭐ |
| sde-find-answer-in-codebase-3 | 0.2 | 0.2 | 0 |
| admin-check-employees-budget-and-reply-and-record | 0.17 | 0.17 | 0.17 |
| admin-collect-requests-and-compute-total-price | 0.12 | 0.12 | 0.25 |
| finance-r-d-activities | 0.08 | 0.08 | 0 |
| hr-mass-survey | 0.07 | 0.07 | 0.07 |
| finance-apply-tax-credit | 0 | 0 | 0 |
| hr-pick-interviewer-3 | 0 | 0 | 0.12 |
| sde-create-new-gitlab-project-logo | 0 | 0 | 0.33 |
| sde-debug-crashed-server | 0.12 | 0 | 0.12 |
| sde-report-agent-repos | 0 | 0 | 0 |
Average Performance Metrics:
| LLM-as-Colleague Model | Resolved Instances | Avg. Steps | Avg. Cost ($) |
|---|---|---|---|
| GPT-4o | 18 | 33.6 | 4.98 |
| DeepSeek v3 | 16 | 30.4 | 4.05 |
| Claude Sonnet 3.7 | 17 | 32.4 | 5.09 |
Several tasks were successfully solved (Score = 1.00) by one or two models but not by all three.
Solved by GPT-4o & DeepSeek v3 (but not Claude Sonnet 3.7)
- admin-ask-for-upgrade-reimbursement (Agent failure. Agent failed before talking to Colleague, likely due to variation)
- hr-collect-multiple-valid-passwords (Agent failure. Environmental issue with this run)
- hr-new-grad-job-description-3 (Agent failure. Agent failed to close popup on ownCloud, likely due to variation)
Solved by Claude Sonnet 3.7 & GPT-4o (but not DeepSeek v3)
- hr-check-for-invalid-passwords-and-ask-for-valid-passwords
DeepSeek v3's response doesn't properly quote the password string, and confused the agent that the line ending character "dot" is part of the password. I am leaning towards that this failure can be attributed to LLM-as-colleague weakness.
- hr-delete-and-insert-user (Agent failure. This task was wrongly annotated as with 'LLM-as-colleague', but in fact it doesn't)
- sde-reply-community-issue-by-asking-npc
Solved by Claude Sonnet 3.7 & DeepSeek v3 (but not GPT-4o)
- sde-create-new-repo
- pm-ask-for-issue-and-create-in-gitlab
- pm-schedule-meeting-2
Solved only by GPT-4o
- hr-new-grad-job-description-2 (Agent failure. Agent failed to close popup on ownCloud).
- pm-prepare-meeting-with-customers
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Claude 3.7 Sonnet experiment results and the GPT-4o and DeepSeek v3 experiment references. The issue presents completed comparisons but does not name a file, test, requested change, or completion criteria, so the intended definition of done needs clarification before work can begin.
Written by the indexing model from the issue text.
Assessment
- Domain
- analytics, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100