Version hashes won't match after spec changes

Lightning and the CLI have the ability to generate short version hash to describe the state of a particular workflow. This is really important for detecting divergence across workflows (if I merge this workflow, will I lose any changes?)

The hash is generated by taking keys from the workflow and building a string with their values. For example, the id and expression and adaptor for each step.

Sometimes we update the workflow schema by adding new keys (for example, we recently added webhook_response_config). When we do this, and re-generate the hash, we get a new value for the same workflow.

This is very problematic when comparing versions both on the app, the CLI, and indeed across both. Without changing the actual workflow, the hash values might change.

It's a particular problem when using an older CLI version. Lightning might start returning a different hash for the same workflow - suddenly it looks like there's been a change, when there hasn't.

Also the workflow history gets invalidated. If two hashes were generated on the same version then OK, they can be compared even when latest changes. But an older hash with the same workflow might look different when comparing to latest main.

So it's a big tricky - the upshot of which is that we'll be giving users a false and scary warning that they're about to lose changes.

This is why https://github.com/OpenFn/kit/issues/1269 keeps popping up. It's because version hashes are falsely mismatching.

What might we do about this?

• Can we use dates somehow? I keep coming back to this. But content is more important than last-touched-date.
• Can we use a deep diff as a differentiator? Don't trust divergence warnings but deep diff instead. But this only works when comparng latest:latest - it doesn't work for history.
• Include a schema version number in the hash, like cli:4.0.1:abcefqg. Now when comparing two hashes, we know the schema they were recorded against. But we can't do anything with this information! All we know is that the versions cannot be compared. If we skip divergence testing, we might overwrite changes. If we warn, we're warning falsely. We can't restore the actual workflow contents from the hash so we can't deep diff. All we can do is say "hey, we can't tell if this change will result in lost changes, so maybe double check". Which is pretty weak.

Maybe if we cannot compare on versions, but we had a date, we could say "we can't diff the target, but we know it was changed on may 12, and this current version was forked on may 11th - so there may be a conflict". It would be a secondary test.

So let's spike a solution like cli:timestamp:abc.

Start by tracing how Lightning and the CLI generate and compare workflow version hashes. Review the effects of schema changes, older CLI versions, and workflow history on divergence warnings. A completed spike should establish whether a timestamp-based hash or another approach can avoid false mismatches while still identifying possible conflicts.