hashicorp / hashicorp/nomad

external plugins exit twice during client startup (EOF)

Open
#25,115 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

hcc/jira stage/needs-investigation theme/driver type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version
Nomad v1.9.6
BuildDate 2025-02-11T18:55:10Z
Revision 7f8b44963d36d025520348d7f24735774d26f13b+CHANGES
Operating system and Environment details

Fedora release 41 (Forty One)
Linux version 6.12.11-200.fc41.x86_64

Issue

When downloading a plugin and installing it in /opt/nomad/plugins/, the plugin is detected but reports error reading from server: EOF.
This has been tested and observed with the exec2 and podman plugins. The same error is seen whether the prebuilt binaries or locally-build versions are used.

The error occurs prior to any jobs being run, on a fresh installation.

2025-02-14T09:44:36.320+1100 [DEBUG] agent.plugin_loader: waiting for RPC address: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2
2025-02-14T09:44:36.534+1100 [DEBUG] agent.plugin_loader.nomad-driver-exec2: plugin address: plugin_dir=/opt/nomad/plugins address=/tmp/plugin3126122633 network=unix timestamp="2025-02-14T09:44:36.534+1100"                                                                             
2025-02-14T09:44:36.534+1100 [DEBUG] agent.plugin_loader: using plugin: plugin_dir=/opt/nomad/plugins version=2
2025-02-14T09:44:36.543+1100 [DEBUG] agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/opt/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"
2025-02-14T09:44:36.546+1100 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2 id=17427
2025-02-14T09:44:36.546+1100 [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/opt/nomad/plugins
Reproduction steps

Install nomad on a fresh fedora 41 installation
Download and install the exec2 plugin (or podman)
Run sudo nomad agent -dev -plugin-dir='/opt/nomad/plugins'

Expected Result

Nomad does not report agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/opt/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"

Actual Result

Nomad reports agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/opt/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"

Job file (if appropriate)
Nomad Server logs (if appropriate)
$ sudo nomad agent -dev -plugin-dir='/opt/nomad/plugins' 
==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
                Client: true
             Log Level: DEBUG
               Node Id: ce3d4ada-ca0d-3a12-d74a-c1b7ab7056d2
                Region: global (DC: dc1)
                Server: true
               Version: 1.9.6

==> Nomad agent started! Log data will stream in below:

    2025-02-14T10:17:39.270+1100 [DEBUG] nomad: issuer not set; OIDC Discovery endpoint for workload identities disabled
    2025-02-14T10:17:39.271+1100 [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:28d1c2ff-a836-5c5b-b1e1-540061459da0 Address:127.0.0.1:4647}]"
    2025-02-14T10:17:39.271+1100 [INFO]  nomad.raft: entering follower state: follower="Node at 127.0.0.1:4647 [Follower]" leader-address= leader-id=
    2025-02-14T10:17:39.271+1100 [INFO]  nomad: serf: EventMemberJoin: 857pt7csjr3hksp2fqgh.ip6.superloop.au.global 127.0.0.1
    2025-02-14T10:17:39.271+1100 [INFO]  nomad: starting scheduling worker(s): num_workers=8 schedulers=["service", "batch", "system", "sysbatch", "_core"]
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=aa310b40-fdb8-c2ad-5786-75b685b0d9f4 index=1 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=30bafb37-da6b-7e49-5ae7-b29ea67c099c index=2 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=aa310b40-fdb8-c2ad-5786-75b685b0d9f4
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=17d4c8ef-2d2e-9725-7373-3970ff79e099
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=30bafb37-da6b-7e49-5ae7-b29ea67c099c
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=17d4c8ef-2d2e-9725-7373-3970ff79e099 index=3 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=69579bbd-2750-3c18-d5cd-02f91bda6a56 index=4 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=3cf035f8-d9bc-c8ac-3b9b-0d61a14d645c index=5 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=13757e33-7a06-ed0b-afc1-40335e993514 index=6 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=6a6a6eab-9c7f-4fd4-a069-0bb721fd7d35 index=7 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=3cf035f8-d9bc-c8ac-3b9b-0d61a14d645c
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=24dc46db-b1d2-7a23-02af-2731f255b3f0
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=13757e33-7a06-ed0b-afc1-40335e993514
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=69579bbd-2750-3c18-d5cd-02f91bda6a56
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad: started scheduling worker: id=24dc46db-b1d2-7a23-02af-2731f255b3f0 index=8 of=8
    2025-02-14T10:17:39.271+1100 [DEBUG] worker: running: worker_id=6a6a6eab-9c7f-4fd4-a069-0bb721fd7d35
    2025-02-14T10:17:39.271+1100 [INFO]  nomad: started scheduling worker(s): num_workers=8 schedulers=["service", "batch", "system", "sysbatch", "_core"]
    2025-02-14T10:17:39.271+1100 [INFO]  nomad: adding server: server="857pt7csjr3hksp2fqgh.ip6.superloop.au.global (Addr: 127.0.0.1:4647) (DC: dc1)"
    2025-02-14T10:17:39.271+1100 [DEBUG] nomad.keyring.replicator: starting encryption key replication
    2025-02-14T10:17:39.271+1100 [WARN]  agent.plugin_loader: plugin not referenced in the agent configuration file, future versions of Nomad will not load this plugin until the agent configuration is updated: plugin_dir=/opt/nomad/plugins plugin=nomad-driver-exec2
    2025-02-14T10:17:39.271+1100 [DEBUG] agent.plugin_loader: starting plugin: plugin_dir=/opt/nomad/plugins path=/opt/nomad/plugins/nomad-driver-exec2 args=["/opt/nomad/plugins/nomad-driver-exec2"]
    2025-02-14T10:17:39.272+1100 [DEBUG] agent.plugin_loader: plugin started: plugin_dir=/opt/nomad/plugins path=/opt/nomad/plugins/nomad-driver-exec2 pid=49377
    2025-02-14T10:17:39.272+1100 [DEBUG] agent.plugin_loader: waiting for RPC address: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2
    2025-02-14T10:17:39.279+1100 [DEBUG] agent.plugin_loader: using plugin: plugin_dir=/opt/nomad/plugins version=2
    2025-02-14T10:17:39.279+1100 [DEBUG] agent.plugin_loader.nomad-driver-exec2: plugin address: plugin_dir=/opt/nomad/plugins address=/tmp/plugin2742831925 network=unix timestamp="2025-02-14T10:17:39.279+1100"
    2025-02-14T10:17:39.281+1100 [DEBUG] agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/opt/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"
    2025-02-14T10:17:39.282+1100 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2 id=49377
    2025-02-14T10:17:39.282+1100 [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/opt/nomad/plugins
    2025-02-14T10:17:39.282+1100 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/opt/nomad/plugins
    2025-02-14T10:17:39.282+1100 [DEBUG] agent.plugin_loader.raw_exec.id_validator: user range configured: plugin_dir=/opt/nomad/plugins denied range=""
    2025-02-14T10:17:39.282+1100 [DEBUG] agent.plugin_loader.raw_exec.id_validator: group range configured: plugin_dir=/opt/nomad/plugins denied range=""
    2025-02-14T10:17:39.282+1100 [DEBUG] agent.plugin_loader: starting plugin: plugin_dir=/opt/nomad/plugins path=/opt/nomad/plugins/nomad-driver-exec2 args=["/opt/nomad/plugins/nomad-driver-exec2"]
    2025-02-14T10:17:39.283+1100 [DEBUG] agent.plugin_loader: plugin started: plugin_dir=/opt/nomad/plugins path=/opt/nomad/plugins/nomad-driver-exec2 pid=49389
    2025-02-14T10:17:39.283+1100 [DEBUG] agent.plugin_loader: waiting for RPC address: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2
    2025-02-14T10:17:39.290+1100 [DEBUG] agent.plugin_loader: using plugin: plugin_dir=/opt/nomad/plugins version=2
    2025-02-14T10:17:39.290+1100 [DEBUG] agent.plugin_loader.nomad-driver-exec2: plugin address: plugin_dir=/opt/nomad/plugins address=/tmp/plugin2320074636 network=unix timestamp="2025-02-14T10:17:39.289+1100"
    2025-02-14T10:17:39.291+1100 [DEBUG] agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/opt/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"
    2025-02-14T10:17:39.291+1100 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/plugins plugin=/opt/nomad/plugins/nomad-driver-exec2 id=49389
    2025-02-14T10:17:39.291+1100 [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/opt/nomad/plugins
    2025-02-14T10:17:39.292+1100 [DEBUG] agent.plugin_loader.exec.id_validator: user range configured: plugin_dir=/opt/nomad/plugins denied range=""
    2025-02-14T10:17:39.292+1100 [DEBUG] agent.plugin_loader.exec.id_validator: group range configured: plugin_dir=/opt/nomad/plugins denied range=""
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=exec2 type=driver plugin_version=v2.0.0
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2025-02-14T10:17:39.292+1100 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
...
Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Nomad's agent.plugin_loader startup path and reproduce with Nomad v1.9.6 on Fedora 41 using the provided exec2 or podman plugin and -plugin-dir. Trace the plugin's RPC and stdio lifecycle around the reported EOF and process exit. Done means external plugins remain available after startup without the repeated EOF and exit logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.