Keychain credentials do not resolve on projects with zero-persistence enabled

What's broken

Keychain credentials do not resolve on a project with zero-persistence turned on. Both features work on their own and break together, so a project that needs both cannot use either properly. Reproduced on v2.18.0.

A keychain does not read the input we hand to the worker. It runs its own lookup against the stored copy of that input in the database. Zero-persistence erases that stored copy inside the same handler that sends the input to the worker, and the worker asks for the input before it asks for any credential. So the keychain ends up matching its path against an empty document.

Two outcomes, and the quiet one is worse. With no default credential, resolution succeeds with no credential at all and the step fails deep inside the adaptor with an authorization error, with nothing on our side logging the miss. With a default credential set, every run silently falls back to the default and the path is ignored, which for the fan-out case keychains exist to serve means every tenant's data going wherever the default points, with no error anywhere.

It only bites when the run's input is a webhook, cron or step-result dataclip with no name. Manual runs off a named or global dataclip are untouched, which is why this passes a hand test and fails in production.

What to fix

Resolve the keychain when we build the run, before anything is wiped, and hand the worker a concrete credential id like we do for every other credential. The worker asks for the plan before it asks for the input, so the lookup would happen while the stored copy is still there. Zero-persistence is untouched and credential selection stays deterministic and auditable per run.

The alternative is to leave the lookup where it is and move the delete to the end of the run. Smaller, and it does fix this, but it keeps the payload in the database for the length of the run rather than milliseconds, and a run that dies in a crash would never be cleaned up at all, since zero-persistence projects have no retention period set and the nightly purge skips them.

Either way it should land with the test that is missing today: a keychain on a project with history retention disabled, fetching the input and then the credential in that order, asserting the right credential comes back rather than the default.

Related

#5051 is a second keychain problem underneath this one, where the path matches the raw webhook payload rather than the wrapped shape the inspector displays. Anyone who wrote their path off the inspector will still miss after this is fixed, so the two are worth taking together.

Relevant code

Lightning.Credentials.Resolver.find_credential_by_jsonpath/2 evaluates the path against Dataclip.body for run.dataclip_id. The wipe happens in the fetch:dataclip handler in LightningWeb.Channels.RunChannel, via Lightning.Runs.wipe_dataclips/1, guarded on the run's save_dataclips option, which is false exactly when the retention policy is erase_all. In @openfn/ws-worker the credential resolver is only reachable inside engine.execute, which is awaited behind the dataclip load. The credential id handed to the worker is built in Lightning.Runs.RunWithOptions.

Start with Lightning.Runs.RunWithOptions and trace how the fetch:dataclip handler in LightningWeb.Channels.RunChannel invokes Lightning.Runs.wipe_dataclips/1. Then inspect Lightning.Credentials.Resolver.find_credential_by_jsonpath/2 and the worker's engine.execute order. Done means a regression test covers erase_all input followed by keychain resolution and confirms the selected credential, not the default, is returned.