Making future types renameable

Candid has support for “future types”, meaning that new versions of Candid can introduce new types in a way that old clients can safely skip over them. Here the assumption is that old clients never care about decoding such data, but due to subtyping may have to skip over them in extra record fields, or in optional types. Therefore, the Candid binary format includes the necessary information:

These measures allow the serialisation format to be extended with new types in the future, as long as their representation and the representation of the corresponding values include a length prefix matching the above scheme, and thereby allowing an older deserialiser not understanding them to skip over them. The subtyping rules ensure that upgradability is maintained in this situation, i.e., an old deserialiser has no need to understand the encoded data.

It seems, however, that we may need old clients to do more than just skip data, if we want to implement generic data or closures (https://github.com/dfinity/candid/issues/245, https://github.com/dfinity/candid/pull/291, https://github.com/dfinity/candid/pull/292). This also comes up with the IC’s HTTP Gateway Protocol, that is specified to pass a generic “token” of any Candid type back to the backend canister.

In these discussion we noticed that currently, such generic, opaque use of Candid values is not possible due to Future Types: If the generic value is such a future type, an old client only knows the size of the value, but doesn’t know the structure of the type description, and thus can’t copy it into a new message.

Luckily, we don’t have any future types yet, and we can refine the spec to make that possible:

• We change

  Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count.

  to something to the effect of

  Any such opcode is followed by an LEB128-encoded count, and then a number of bytes corresponding to this count. These bytes start with a LEB128-encoded count, followed by that many SLEB-128 encoded type indices (negative for primitive types, positive for type table references). The remaining data in this type description may not contain any type table indices.

• We also say somewhere that the semantics of a future type must not be affected by renumbering the type table entries, or merging identical table entries.

This way, a client can generically store a value together with its (portion of the) type table, and insert the value anywhere in another Candid value, correctly merging the type tables.

And it’s even backward compatible!

WDYT, @rossberg?

Start by reading the Candid specification passages on future-type opcodes, type descriptions, and type-table indices. Review the linked discussions in issues 245 and pull requests 291 and 292, along with the IC HTTP Gateway Protocol reference. Done means an agreed, backward-compatible specification refinement that permits generic values to retain and relocate the needed type information.