Allow any JSON value for dataclips

Currently, Lightning only accepts JSON objects as dataclips. This was the way it worked from way back in the day ({} and { data: {...} } being the primordial "initial states" for runs.

Could we allow other JSON values, such as 42, true, or null? It sure would be intuitive to start a cron-triggered run with null if you didn't need any pre-existing state.

My guess, however, is that most/many/several/some adaptors rely on state as being an object (they push new keys into it, which you can't do with null).

So do we want to accept anything other than an object for input/output state? And if we do... what breaks? What contracts (likely some around adaptors, possibly also in several places around dataclips in Lightning) would need re-authoring?

Thank you @josephjclark and @stuartc for raising this. Wanted to pop it into GitHub for posterity. Joe's description below:

Should dataclips emitted by Lightning be JSON Objects, or any JSON values?

In practice I think 99.99% of all runs ever (on v1 or v2 platforms) have probably returned JSON Objects.

The runtime is designed to accept any JSON value. It's just a data pipeline and doesn't have strong opinions about what the result should look like - and I think that's correct.

There's no reason why Lightning has to mirror this rule, I guess, although it's obviously nice if the CLI and app behave the same way.

If we allow any JSON value to be a dataclip, what breaks in the app?

If we really really want to force a dataclip to be an object, should we impose the same constraints on the worker? On the runtime?

This relates to #4627 and #4626

The issue names dataclips, adaptors, and the runtime as the relevant boundaries, but no files or tests. Start by locating where Lightning validates dataclip input and output state, then inspect adaptor contracts for object-only assumptions. Done means documenting the compatibility impact and reaching a decision on whether non-object JSON values are supported.