livepeer / livepeer/go-livepeer

Remote signer: live session pays once and is then released for insufficient balance

Open
#4,077 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: triage
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

Describe the bug

A live session funded through -remoteSigner mints tickets on its first /generate-live-payment call and then never mints again. Every subsequent call is answered HTTP 482 no tickets. The session runs normally until the credit from that one opening payment is spent, then the orchestrator releases it with insufficient balance. Nothing in the signer's logs looks like a failure — each cycle reads as a routine skip.

The remote signer is stateless by design: the session balance and the clock live in a signed RemotePaymentState that travels with the client, and the signer rebuilds sessionBalance from state.Balance on each request. The no tickets branch in server/remote_signer.go returns a bare error and no state. Its comment says the caller "should retry once balance has been run down further", but the caller has no way to run it down — Balance.Reserve() has already zeroed the in-memory balance, completeBalanceUpdate is never reached, and nothing new is handed back — so the client resends the same state.Balance and the same state.LastUpdate, and the signer computes the same answer for ever.

That branch is not an edge case. Immediately after any successful payment the client holds existingCredit = numTickets × ev, while the threshold is minCredit = max(fee, ev) (newBalanceUpdate raises minCredit to the ticket EV when the fee is below it). Since numTickets = ceil(creditGap / ev) ≥ 1, the relation existingCredit ≥ minCredit holds unconditionally, for any price and any ticket EV. So the very first cycle after the opening payment takes this branch, and so does every cycle after that.

To Reproduce

Steps to reproduce the behavior:

  1. Run an orchestrator with a metered live runner (BYOC), priced per second. Ours: -liveRunnerConfig runners.json with "price_info": {"price": 0.5, "currency": "usd", "unit": "hour"}, and -ticketEV 10000000000000.
  2. Run a gateway against it with -remoteSigner, using a client that funds on a timer — e.g. livepeer-python-gateway v1.0.0, whose LivePaymentSession.run_payments() calls /generate-live-payment every PAYMENT_INTERVAL_S = 3.0.
  3. Open a live session and keep it open for longer than ticketEV / pricePerSecond. At the values above that is about 176 seconds.
  4. See the session released at insufficient balance, having received exactly one payment.

Expected behavior

The payment loop should keep the session funded for as long as the client keeps paying. A cycle where no tickets are needed should still advance the session — debit the service consumed since LastUpdate and hand the caller the updated state — so that the balance runs down and the next cycle mints when it should. Instead the session is funded once and then coasts to a stop.

Screenshots

Not applicable — server-side. Logs instead.

Orchestrator, session priced at 0.5 usd/hour with -ticketEV 10000000000000:

17:10:53 E ai_http.go:350] runner_id=donkey-push session_id=3b40b139 Error accounting live runner payment, releasing session err=insufficient balance, mid=3b40b139, fee=283759014725, balance=68434484625
17:10:53 I live runner session released runner_id=runner_vkab3uva session_id=3b40b139 app=donkey-push duration=3m0.001693228s

Gateway, every cycle for the whole three minutes, thirty of them, identical:

DEBUG Payment loop skipped cycle: Signer returned HTTP 482 (skip payment cycle) (url=http://127.0.0.1:7936/generate-live-payment); body='no tickets'

The orchestrator logged Processing live payment every 5s throughout and received zero payments after the first. The arithmetic matches the release time to the second: the opening payment was one ticket at EV 1e13, the measured burn was 5.68e10 wei/s, and 1e13 / 5.68e10 = 176s against an observed 180s.

Desktop (please complete the following information):

Not applicable — no browser involved. Server environment instead:

  • OS: Ubuntu 24.04 LTS (both hosts), amd64
  • go-livepeer: v0.9.2, built -tags mainnet, Arbitrum mainnet
  • Client: livepeer-python-gateway v1.0.0 on Python 3.12

Smartphone (please complete the following information):

Not applicable.

Additional context

server/remote_signer.go is byte-identical at v0.9.1, v0.9.2 and on master (c1bd7495), so this is not specific to the version we happen to run.

-remoteSigner appears to be the only path through this code, which is probably why it has gone unreported. A gateway that holds its own key keeps sess.Balance in memory across segments, so Reserve() and completeBalanceUpdate() pair up correctly inside one process and the balance does run down. The failure only exists because the remote signer externalises that state and one path forgets to hand it back.

It is also easy to mistake for a tuning problem. Raising -ticketEV enlarges the single opening payment, so the session survives proportionally longer — which reads exactly like a fix until a longer run is attempted. We chased it that way twice, and short runs (20s, 60s, 90s) all passed because they fit inside the opening credit.

Topology, in case it matters: orchestrator and gateway on separate hosts, remote signer bound to loopback on the gateway host, no webhook auth on the signer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in server/remote_signer.go and trace the no-tickets branch from /generate-live-payment through RemotePaymentState, Balance.Reserve(), and completeBalanceUpdate. Reproduce the repeated HTTP 482 behavior, then verify that a no-ticket cycle advances the signed state and that later cycles mint tickets as the balance is consumed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, payments
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.