inputs seem to have not been input

I'm a few different kinds of confused, and I'm not sure how much is me holding buildomat wrong versus buildomat also being confused, and so I'm hoping to take some notes here and work that into more useful tickets for everyone involved as appropriate. I'm extremely open to pointers about what I'm misunderstanding, too, please!!

the jobs relevant (to my knowledge) are Propolis' phd-build and test-gimlet. the former builds some test artifacts, the latter runs them in one of the places we don't really want to go setting up toolchains (e.g. a gimlet). these haven't changed in months, and certainly not in the branches I'm looking at below.

this generally works fine. here's a recent test-gimlet job that looks normal to me, from a few days ago: [bmat log]

/input/phd-build.out/propolis-tests-debug.tar.gz clearly exists, we can see the downloading/downloaded lines above, the tar has been copied around and extracted and tests ran. great all around.

so, here's a recent Propolis test-gimlet job: [bmat log]. there are no downloading lines! there's also a job store has no value for "GITHUB_TOKEN"; waiting for a value... a bit later. then we discover that, indeed, there is nothing at /input/phd-build/out/propolis-tests-debug.tar.gz and that's the end of it. what gives. the check run there is https://github.com/oxidecomputer/propolis/pull/702/checks?check_run_id=102205427707, and I believe that means the inputs would come from the job at https://github.com/oxidecomputer/propolis/pull/702/checks?check_run_id=93992053972 which is weird I think mostly for having been run about a month prior.

I would have written this off as "maybe the tarballs fell out of a cache or something and I should rerun all of it", but I can still download propolis-tests-debug.tar.gz as with

curl https://buildomat.eng.oxide.computer/wg/0/artefact/01KZSX1TCN1DMJE9VFN45RVGM7/s54V2ABDWunK03p9CRKcMwW8I19AjI5NWevQqSgbQB3yTufj/01KZSX2CDG5ZZZKP1XQEWPHKP8/01KZSXQV30HQP7VEYW451ZWBMP/propolis-tests-debug.tar.gz

so the file is still there in buildomat (which tbh is actually what I'd expected!), leaving me to ask ... why did the agent not download them, and what did it think the inputs are if not the outputs of the dependent job?

I've rerun this failing test-gimlet job via the Re-run link on the GitHub check run page (that is the same link as above, https://github.com/oxidecomputer/propolis/pull/702/checks?check_run_id=102205427707). not that I expect this is useful but on the off chance that's an "interesting" way to get a job rerun somehow, now you know.

I think that what we must find here is that job.inputs reports an empty list for this job id, and that is some kind of a bug. I don't really understand what's going on with GITHUB_TOKEN or how it could be unset here either.

the only thing I can imagine here is if that by rerunning a job that was initially created a month ago, typically we'd I'd be relying on in-server state connecting the job ID to a store with a token and with inputs looked up, but that perhaps the server has restarted between now and then and everything draws a blank?

Start with server/src/api/worker.rs lines 202-212 and compare the affected job's reported inputs with the normal test-gimlet log and the linked phd-build artefact. Reproduce or trace the rerun behavior, including the missing GITHUB_TOKEN, and consider the issue done when the cause of the empty inputs and missing downloads is identified and corrected.