Solving flat namespace issue (@joystream/types)

The problem

Quite a few times we ran into a problem where types with the same name, beeing part of different modules in the runtime, were causing issues due the way the runtime metadata is generated and then parsed by the @polkadot/api library.

In case there are two types, beeing part of two different modules in the runtime (ie. ThreadId type beeing part of both proposalDiscussion and forum module) - we cannot register them as serparate types using @polkadot/api, unless they have separate names in the metadata.

This metadata (which is returned by api.rpc.state.getMetadata() method) basically describes the types of: items in the exposed storage (available through api.query methods), arguments of exposed extrinsics (api.tx) and the events data. Part of it can look like this:

"metadata": {
    "V8": {
        "modules": [
            {
                "name": "ModuleA",
                "storage": {
                    "prefix": "ModuleA",
                    "items": [
                        {
                            "name": "TotalModuleABalance",
                            "modifier": 1,
                            "type": "CustomBalanceType"
                            "fallback": "0x00000000000000000000000000000000",
                            "documentation": [
                                "Some total balance tracked by module A."
                            ]
                        },
                    ],
                },
                "calls": [
                    {
                        "name": "slash_balance",
                        "args": [
                            {
                                "name": "amount",
                                "type": "CustomBalanceType"
                            }
                        ]
                        /*...*/
                    }
                ],
                "events": /* ... */
            },
            {
                "name": "ModuleB",
                "storage": {
                    "prefix": "ModuleB",
                    "items": [
                        {
                            "name": "TotalModuleBBalance",
                            "modifier": 1,
                            "type": "CustomBalanceType"
                            "fallback": "0x00000000000000000000000000000000",
                            "documentation": [
                                "Some total balance tracked by module B."
                            ]
                        },
                    ],
                },
                "calls": [
                    {
                        "name": "slash_balance",
                        "args": [
                            {
                                "name": "amount",
                                "type": "CustomBalanceType"
                            }
                        ]
                        /*...*/
                    }
                ],
                "events": /* ... */
            },
            /*...*/
        ]
    }
}

In that case both ModuleA and ModuleB have storage items of type CustomBalanceType and extrinsics (calls) which take CustomBalanceType as arguments (named slash_balance).

The problem is that even though CustomBalanceType may be completely different in ModuleA and ModuleB (due to different Rust namespaces), it is considered to be the same type inside the metadata. It causes the @polkadot/api to expect us to only register one CustomBalanceType.

In case we register CustomBalanceType inside @joystream/types based on the implementation from runtime ModuleA - there will be issues with data enconding and decoding when we use api.query or api.tx for ModuleB (and vice-versa). This may cause the client applications to either break, send incorrect data to the runtime or get incorrect values when reading from the storage.

Potential solution

In order to solve this we could override each occurrence of CustomBalanceType inside each module metadata with [ModuleName]::CustomBalanceType. We could create a script that would do that for all the conflicting types or for all custom types in general. While implementing this we'd have to be careful not to override the native substrate/rust types like Balance, u32 or Moment though. One way to solve this would be by creating a list of types for which the namespaces should be expected, or on the contrary - the list of types for which the namespace shouldn't be expected (perhaps there exist some better, alternative solutions that may be implemented here).

The overriden metadata can be passed as an arugment when intiailizng the Api instance, ie.:

import { ApiPromise, WsProvider } from '@polkadot/api';
import MetadataVersioned from '@polkadot/types/Metadata/MetadataVersioned';
import { registerJoystreamTypes } from '@joystream/types';

export async function initApi(apiUri: string) {
    const initWsProvider = new WsProvider(apiUri);
    registerJoystreamTypes();
    const api = await ApiPromise.create({ provider: initWsProvider });

    const [genesisHash, runtimeVersion] = await Promise.all([
        api.rpc.chain.getBlockHash(0),
        api.rpc.state.getRuntimeVersion()
    ]);
    // Create metadata key required by @polkadot/api
    const metadataKey = `${genesisHash}-${runtimeVersion.specVersion}`;
    const originalMetadata = await api.rpc.state.getMetadata();
    const metadataJson = originalMetadata.toJSON();
    // ...
    // Here we can override the types in the metadataJson object
    // ...
    const overridenMetadata = new MetadataVersioned(metadataJson);
    const metadataArg = { [metadataKey]: overridenMetadata.toHex() };

    const wsProvider = new WsProvider(apiUri);

    return await ApiPromise.create({ provider: wsProvider, metadata: metadataArg });
}

Start by tracing registerJoystreamTypes in @joystream/types and the ApiPromise initialization flow using api.rpc.state.getMetadata(). Read the MetadataVersioned handling shown in the issue before deciding how metadata names should be transformed. Done means conflicting runtime type names are distinguishable without incorrectly namespacing native types, with the resulting metadata accepted by @polkadot/api.