Charging for Failed Tool Calls
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
- Copilot Chat Extension Version: All
- VS Code Version: All
- OS Version: All
- Feature (e.g. agent/edit/ask mode): Tool Calls
- Selected model (e.g. GPT 4.1, Claude 3.7 Sonnet): All
- Logs: All
Steps to Reproduce:
It really sucks that GitHub/VSCode is charging for failed tool calls. It seems this sets up a negative incentive for increasing the number of failed tools, which result in re-trying many times with either other tools, or with different input. This burns tokens at a very high rate...especially for big changes...
It would be good if:
1) we track and display to the user which tools are being called, and how often these calls fail
2) work to minimize these by fixing issues, and improving the tooling so that it fails less
3) optionally do not charge for tokens when there are significant numbers of tool failures (especially the default/built-in ones), or provide credits (maybe at 50%)
4) when tools are regularly failing, disable them, or look for alternatives to enable
Contributor guide
Assessment
This issue has not been assessed yet.