BYOK is broken in a fatal way!
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 2.1k
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
BYOK is broken in a fatal way!
While I was trying out GitHub Copilot App with BYOK against an local endpoint which is supported by VSCode (so the endpoint should be working just fine), at the start of an session I received an error stating falling back to auto because BYOK was not available, which is bad in itself but okay.
Still I immediately pressed abort, switched back in the model selection from "auto" to my local LLM provider, then resent my message. Then I saw no error and everything seemed to be working fine now. (I wanted to try it out again because it is not uncommon for my local LLM endpoint to be sometimes not available for a few seconds because of live adjustments.)
But now after several new messages submitted with still my LLM provider selected, I suddenly receive a message with "You've used up all of your AI credits.", are you kidding me ?!?
I even checked the integrated token usage bar and it is STILL showing 1% used! While the GitHub web UI seems to correctly show that used up all my tokens for this month. Form the usage dashboards of my local LLM provider I can confirm that GitHub Copilot App has actually used its Cloud models because no usage was logged locally in the last hour. But the UI of the app is just plain lying, I cannot state it in a better way!
| Field | Value |
|---|---|
| App version | 1.1.17 |
| OS | Windows 10.0.26200 |
| Theme | GitHub |
| Path | /chat |
| Tenure | Day 1 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce this in GitHub Copilot App 1.1.17 on Windows using a VSCode-compatible local endpoint: make the endpoint unavailable, observe fallback to auto, then switch back to the local provider and send messages. Compare the selected provider, integrated token bar, GitHub web usage, and local provider logs. Done means BYOK failures do not silently consume cloud credits and the app reports usage accurately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, vscode
- Domain
- ai, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100