github / github/app

BYOK is broken in a fatal way!

Open
#3,731 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
2.1k
Forks
157
PR merge metrics
No merged PRs in 30d

Description

BYOK is broken in a fatal way!
While I was trying out GitHub Copilot App with BYOK against an local endpoint which is supported by VSCode (so the endpoint should be working just fine), at the start of an session I received an error stating falling back to auto because BYOK was not available, which is bad in itself but okay.
Still I immediately pressed abort, switched back in the model selection from "auto" to my local LLM provider, then resent my message. Then I saw no error and everything seemed to be working fine now. (I wanted to try it out again because it is not uncommon for my local LLM endpoint to be sometimes not available for a few seconds because of live adjustments.)
But now after several new messages submitted with still my LLM provider selected, I suddenly receive a message with "You've used up all of your AI credits.", are you kidding me ?!?
I even checked the integrated token usage bar and it is STILL showing 1% used! While the GitHub web UI seems to correctly show that used up all my tokens for this month. Form the usage dashboards of my local LLM provider I can confirm that GitHub Copilot App has actually used its Cloud models because no usage was logged locally in the last hour. But the UI of the app is just plain lying, I cannot state it in a better way!


Field Value
App version 1.1.17
OS Windows 10.0.26200
Theme GitHub
Path /chat
Tenure Day 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce this in GitHub Copilot App 1.1.17 on Windows using a VSCode-compatible local endpoint: make the endpoint unavailable, observe fallback to auto, then switch back to the local provider and send messages. Compare the selected provider, integrated token bar, GitHub web usage, and local provider logs. Done means BYOK failures do not silently consume cloud credits and the app reports usage accurately.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, vscode
Domain
ai, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.