smartcontractkit / smartcontractkit/chainlink

[NODE] Missing job requests.

Open
#6,012 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

investigating
Dominant language
Go
Stars
8.2k
Forks
2k
Avg merge
1d 23h
Merged PRs (30d)
202

Description

Description
The node receives heads but missing jobs.

Basic Information
While running release V1.1.0 the following occurred twice. The node was set up with a minimal env file specifying only the bare minimum and setting the chain nodes etc through the GUI to be round robined. The node will run fine for a while but suddenly will start missing jobs. The logs display the node receiving heads and processing them from multiple endpoints, in this case, 4 different RPC/wss connections. This will result in LINK being locked in the oracle contract and jobs staying unfulfilled. There are no errors in the logs and the only way to spot these malfunctions is constantly monitoring the incoming and outgoing transactions for the oracle.

  • Network: Polygon
  • Operating System: Docker Ubuntu 20.04 LTS
  • Commit: v1.1.0
  • Hosting Provider: self-hosted
  • Startup Command: cd ~/.chainlink && sudo docker run --network chainlink --name chainlink --restart unless-stopped -p X:X -v ~/.chainlink:/chainlink -it --env-file=.env smartcontract/chainlink:1.1.0 local n -p /chainlink/.psw/.password -a /chainlink/.psw/.api

Environment Variables
ROOT=/chainlink
ETH_CHAIN_ID=137
LOG_LEVEL=debug
CHAINLINK_TLS_PORT=
SECURE_COOKIES=true
ALLOW_ORIGINS=*
DATABASE_URL=
DATABASE_TIMEOUT=0
TLS_CERT_PATH=/chainlink/.tls/server.crt
TLS_KEY_PATH=/chainlink/.tls/server.key

Steps to Reproduce
Run the node and get transactions.

Additional Information
No errors are logged. All chain connections seem in sync (4 different endpoints). Node is processing heads. A restart fixes the issue. Can be detected by monitoring incoming link payments vs outgoing tx'es.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the v1.1.0 Docker startup command and minimal environment shown in the issue, then reproduce the node receiving heads from four RPC/WSS endpoints while jobs go missing. Compare incoming heads and LINK payments with outgoing transactions and debug logs; done means the missed-job condition is explained and no longer occurs silently, or produces a useful error signal.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, blockchain, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.