[FEAT] allow JSON with vars inputs in the abi attribute of the ethabiencode2 task

Resources

Chainlink Release Notes v1.1.0 - first mention of ethabiencode2
Chainlink - ethabiencode2 implementation
Chainlink - ethabiencode2 implementation tests
cbor.me - Online CBOR encoder

Description

The abi attribute of the ethabiencode2 task does not allow an input using the $(variable) syntax. It requires a hard coded string, otherwise it errors with:

ETHABIEncode: while parsing ABI string: invalid character '$' looking for beginning of value: bad input for task

Allowed 1:

encode_data    [
                          type="ethabiencode2"
                          abi=<{"inputs":[{"name":"requestId","type":"bytes32"}, {"name":"games","type":"bytes32[]"}]}>
                          data=<{"requestId": $(decode_log.requestId), "games": $(parse_result.games)}>
                         ]

Allowed 2:

encode_data    [
                          type="ethabiencode2"
                          abi=<{"inputs":[{"name":"requestId","type":"bytes32"}, {"name":"timestamp","type":"uint40"},{"name":"statusId","type":"uint8"}, {"name":"games","type":"bytes32[]"}]}>
                          data=<{"requestId": $(decode_log.requestId), "timestamp": $(parse_result.timestamp), "statusId": $(parse_result.statusId), "games": $(parse_result.games)}>
                         ]

Unsupported 1:

encode_data    [
                          type="ethabiencode2"
                          abi=<{"inputs": $(parse_abi.inputs_as_array)}>
                          data=<{"requestId": $(decode_log.requestId), "games": $(parse_result.games)}>
                         ]

Unsupported 2:

encode_data    [
                          type="ethabiencode2"
                          abi="$(parse_abi.inputs_as_json_object)"
                          data=<{"requestId": $(decode_log.requestId), "games": $(parse_result.games)}>
                         ]

Motivation

Lots of successful Direct Request integrations for sports (e.g. therundown, sportsdataio, enetscores) return an array of items (i.e. bytes[], bytes32[]) encoded at EA level (mostly encode packed). Any modification to the existing encoding (e.g. adding/removing a datum/data either on the fulfillment method arguments or the array items) introduces breaking changes. Even though these can be handled versioning the EA, this approach does not solve:

• The contained data (too much data). For instance, an AccuWeather Consumer may be only interested in 2 out of the 12 metrics returned. But the EA returns them all. The same applies to any of the sport integrations; a Consumer may not be interested in the team names (long strings) or may want the scores by period (a uint8[]) instead of the final score (a uint8).
• The encoding. For instance, a more savvy Consumer that requires results with more data may choose encoding packed each array item.

The Allowed 1 and Allowed 2 cases above illustrate two of the many possible encodings the same DR Job can have using ethabiencode2. The problem is that both have the encoding hard coded in the TOML spec. By allowing JSON vars as input in the abi attribute then the Consumer and the NodeOp could define on-demand (per job run) the encoding using any of these strategies:

• The Consumer CBOR encodes the solc ABI (JSON object) and sends it in the Chainlink Request.
• The Consumer CBOR encodes the solc ABI (JSON object) and stores it ina public storage variable. The job run reads it via ethcall.
• If the Consumer does not specify a solc ABI, the NodeOp can always default one by hard coding it in the TOML spec.

Gist of a TOML spec that showcases the hierarchy defined above. On a related note, this is why I asked about how using the cborparse task in standard mode.

I have no idea yet how OCR2DR will work, but I see "dynamic LINK pricing + encode on-demand" like food + wine pairing. And imho the first step is allowing ethabiencode2 to set the abi from a job run var. Moreover, it will ease how EA builders deal with providing more data (it won't be a breaking change, nor will require multiple EAs running) and how Consumers adopt it (at their own pace just by just setting the new encoding).

Personally, I'm interested cause I've built a library called encode on-demand that allows a more granular/deeper encoding (e.g. encode packed/default the items of the array, create on the fly length vars, deal with zero pads for bytesN, etc.) and ideally it sends the resulting standard solc ABI (KO, feature request) & values (OK) to ethabiencode2. The fact that abi does not accept JSON vars as input makes it less powerful/flexible.

Let me know!

Thanks

Start with core/services/pipeline/task.eth_abi_encode_2.go and its tests in core/services/pipeline/task.eth_abi_encode_2_test.go. Trace how the ethabiencode2 task parses its abi attribute and how variables are handled in data. Add coverage for the supported dynamic JSON-variable form while preserving the documented unsupported cases, then run the task tests.