paritytech / paritytech/polkadot-cli

RFC: XCM file format for authoring cross-chain programs

Open
#13 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
10
Forks
2
Avg merge
12h 35m
Merged PRs (30d)
4

Description

Problem

Building XCM programs for dot tx requires inline JSON with deeply nested {type, value} discriminated unions. This is error-prone, unreadable, and impossible to maintain.

Real example from scripts/setup-pusd.sh -- a single reserve transfer:

{"type":"V5","value":[{"type":"SetFeesMode","value":{"jit_withdraw":true}},{"type":"WithdrawAsset","value":[{"id":{"parents":0,"interior":{"type":"X2","value":[{"type":"PalletInstance","value":50},{"type":"GeneralIndex","value":3}]}},"fun":{"type":"Fungible","value":100000000000}}]},{"type":"InitiateTransfer","value":{"destination":{"parents":1,"interior":{"type":"X1","value":[{"type":"Parachain","value":1004}]}},"remote_fees":{"type":"ReserveDeposit","value":{"type":"Definite","value":[{"id":{"parents":0,"interior":{"type":"X2","value":[{"type":"PalletInstance","value":50},{"type":"GeneralIndex","value":3}]}},"fun":{"type":"Fungible","value":1000000000}}]}},"preserve_origin":false,"assets":[{"type":"ReserveDeposit","value":{"type":"Wild","value":{"type":"All"}}}],"remote_xcm":[{"type":"SetFeesMode","value":{"jit_withdraw":true}},{"type":"DepositAsset","value":{"assets":{"type":"Wild","value":{"type":"All"}},"beneficiary":{"parents":0,"interior":{"type":"X1","value":[{"type":"AccountId32","value":{"id":"0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"}}]}}}}]}}]}

XCM is essentially a programming language (XCM docs) with 51 instructions in V5, nested locations, asset types, and sub-programs. It deserves a proper file-based authoring experience.

Proposal

Add support for .xcm.yaml / .xcm.json files that can be passed to dot tx --file or a new dot xcm command.

Scope

v1 (this issue):

  • Read XCM programs from a file with nicer syntax than raw JSON
  • Variables/parameters (optional, nice-to-have)
  • File extension: .xcm.yaml or .xcm.json

Future work (not this issue):

  • REPL with autocomplete from chain metadata (dot xcm --repl)
  • Intent-level abstractions ("teleport 100 DOT to Alice on AssetHub")
  • Linting/validation against chain metadata
  • Dry-run simulation
  • dot xcm as a richer command that checks receiver-side arrival/errors

Ecosystem context

No XCM DSL or file format exists today. The ecosystem has:

  • Raw JSON inline (current approach)
  • ParaSpell SDK -- high-level fluent builder (JS/TS only, intent-level)
  • polkadot-sdk #4736 -- Rust builder proposal, design phase
  • No REPL, no .xcm format, no linter

Format comparison

Four options compared below using the same three example programs.

Reference: example programs in raw JSON
Small: simple teleport (3 instructions) 
{
  "type": "V5",
  "value": [
    {
      "type": "WithdrawAsset",
      "value": [
        {
          "id": { "parents": 1, "interior": { "type": "Here" } },
          "fun": { "type": "Fungible", "value": 1000000000000 }
        }
      ]
    },
    {
      "type": "BuyExecution",
      "value": {
        "fees": {
          "id": { "parents": 1, "interior": { "type": "Here" } },
          "fun": { "type": "Fungible", "value": 1000000000000 }
        },
        "weight_limit": { "type": "Unlimited" }
      }
    },
    {
      "type": "DepositAsset",
      "value": {
        "assets": { "type": "Wild", "value": { "type": "All" } },
        "beneficiary": {
          "parents": 0,
          "interior": {
            "type": "X1",
            "value": [
              {
                "type": "AccountId32",
                "value": { "id": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" }
              }
            ]
          }
        }
      }
    }
  ]
}
Medium: reserve transfer with fees (from setup-pusd.sh) 
{
  "type": "V5",
  "value": [
    { "type": "SetFeesMode", "value": { "jit_withdraw": true } },
    {
      "type": "WithdrawAsset",
      "value": [
        {
          "id": {
            "parents": 0,
            "interior": {
              "type": "X2",
              "value": [
                { "type": "PalletInstance", "value": 50 },
                { "type": "GeneralIndex", "value": 3 }
              ]
            }
          },
          "fun": { "type": "Fungible", "value": 100000000000 }
        }
      ]
    },
    {
      "type": "InitiateTransfer",
      "value": {
        "destination": {
          "parents": 1,
          "interior": {
            "type": "X1",
            "value": [{ "type": "Parachain", "value": 1004 }]
          }
        },
        "remote_fees": {
          "type": "ReserveDeposit",
          "value": {
            "type": "Definite",
            "value": [
              {
                "id": {
                  "parents": 0,
                  "interior": {
                    "type": "X2",
                    "value": [
                      { "type": "PalletInstance", "value": 50 },
                      { "type": "GeneralIndex", "value": 3 }
                    ]
                  }
                },
                "fun": { "type": "Fungible", "value": 1000000000 }
              }
            ]
          }
        },
        "preserve_origin": false,
        "assets": [
          {
            "type": "ReserveDeposit",
            "value": { "type": "Wild", "value": { "type": "All" } }
          }
        ],
        "remote_xcm": [
          { "type": "SetFeesMode", "value": { "jit_withdraw": true } },
          {
            "type": "DepositAsset",
            "value": {
              "assets": { "type": "Wild", "value": { "type": "All" } },
              "beneficiary": {
                "parents": 0,
                "interior": {
                  "type": "X1",
                  "value": [
                    {
                      "type": "AccountId32",
                      "value": {
                        "id": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"
                      }
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  ]
}
Format 1: Pure YAML

Enums as single-key maps (VariantName: value). Void variants as bare strings. Locations as nested objects.

Small: simple teleport 
version: V5
instructions:
  - WithdrawAsset:
      - id: { parents: 1, interior: Here }
        fun:
          Fungible: 1000000000000

  - BuyExecution:
      fees:
        id: { parents: 1, interior: Here }
        fun:
          Fungible: 1000000000000
      weight_limit: Unlimited

  - DepositAsset:
      assets:
        Wild: All
      beneficiary:
        parents: 0
        interior:
          X1:
            - AccountId32:
                id: "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"
Medium: reserve transfer 
version: V5
instructions:
  - SetFeesMode:
      jit_withdraw: true

  - WithdrawAsset:
      - id:
          parents: 0
          interior:
            X2:
              - PalletInstance: 50
              - GeneralIndex: 3
        fun:
          Fungible: 100000000000

  - InitiateTransfer:
      destination:
        parents: 1
        interior:
          X1:
            - Parachain: 1004
      remote_fees:
        ReserveDeposit:
          Definite:
            - id:
                parents: 0
                interior:
                  X2:
                    - PalletInstance: 50
                    - GeneralIndex: 3
              fun:
                Fungible: 1000000000
      preserve_origin: false
      assets:
        - ReserveDeposit:
            Wild: All
      remote_xcm:
        - SetFeesMode:
            jit_withdraw: true
        - DepositAsset:
            assets:
              Wild: All
            beneficiary:
              parents: 0
              interior:
                X1:
                  - AccountId32:
                      id: "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"

Pros:

  • Standard format, well-known, mature parsers
  • Single-key map pattern (Fungible: 1000) is natural for enums
  • Comments with #
  • YAML anchors (&name / *name) provide native variable reuse
  • Good editor support, syntax highlighting out of the box

Cons:

  • Locations still verbose ({parents: 0, interior: {X2: [{PalletInstance: 50}, ...]}})
  • Significant whitespace is error-prone; stray indent changes meaning
  • Deep nesting (5+ levels in medium example) is hard to track visually
  • YAML implicit typing gotchas: 0x00 may parse as 0, yes/no as booleans
  • Needs schema-aware post-processing: single-key maps must be converted to {type, value} objects, but the parser must distinguish enum variants from struct fields -- requires chain metadata
  • New dependency (yaml npm package)

Parse pipeline: YAML.parse() -> recursive walk converting {VariantName: value} maps to {type, value} objects (metadata-aware) -> feed into existing normalizeValue()

Format 2: YAML + shorthand sugar

Same as pure YAML but with inline shorthand for locations, enums, and numeric underscores.

Small: simple teleport 
version: V5
vars:
  alice: "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"

instructions:
  - WithdrawAsset:
      - id: ..
        fun: Fungible(1_000_000_000_000)

  - BuyExecution:
      fees:
        id: ..
        fun: Fungible(1_000_000_000_000)
      weight_limit: Unlimited

  - DepositAsset:
      assets: Wild(All)
      beneficiary: ./AccountId32($alice)
Medium: reserve transfer 
version: V5
vars:
  pusd: ./PalletInstance(50)/GeneralIndex(3)
  alice: "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"

instructions:
  - SetFeesMode:
      jit_withdraw: true

  - WithdrawAsset:
      - id: $pusd
        fun: Fungible(100_000_000_000)

  - InitiateTransfer:
      destination: ../Parachain(1004)
      remote_fees:
        ReserveDeposit:
          Definite:
            - id: $pusd
              fun: Fungible(1_000_000_000)
      preserve_origin: false
      assets:
        - ReserveDeposit(Wild(All))
      remote_xcm:
        - SetFeesMode:
            jit_withdraw: true
        - DepositAsset:
            assets: Wild(All)
            beneficiary: ./AccountId32($alice)

Location shorthand:

Shorthand Expands to
. {parents: 0, interior: Here}
.. {parents: 1, interior: Here}
../.. {parents: 2, interior: Here}
./Parachain(1000) {parents: 0, interior: {X1: [Parachain(1000)]}}
../Parachain(1000)/PalletInstance(50) {parents: 1, interior: {X2: [Parachain(1000), PalletInstance(50)]}}

Enum sugar: Fungible(1000) -> {type: "Fungible", value: 1000}, Wild(All) -> {type: "Wild", value: {type: "All"}}

Pros (over pure YAML):

  • Locations compress dramatically: ../Parachain(1004) vs 4 lines of nested YAML
  • vars block with $name references prevents copy-paste of long hex strings and repeated locations
  • Numeric underscores: 1_000_000_000_000 for readability
  • Enum sugar: Fungible(1000) instead of Fungible: 1000 -- marginal but consistent with Rust syntax

Cons (over pure YAML):

  • Two "languages" to learn: YAML structure + inline shorthand syntax
  • Inline expressions like ReserveDeposit(Wild(All)) with nested parens get fragile
  • Need a string parser for the shorthand (regex + small recursive descent)
  • Harder to validate: YAML parser sees strings, shorthand errors surface later

Parse pipeline: YAML.parse() -> substitute $vars -> walk tree expanding shorthand strings (locations, enums, underscores) -> convert remaining YAML enum maps to {type, value} -> feed into normalizeValue()

Format 3: Pure JSON5/JSONC

Standard {type, value} structure with comments and trailing commas. No shorthand.

Small: simple teleport 
// Simple teleport: withdraw DOT, buy execution, deposit to Alice
{
  "type": "V5",
  "value": [
    // Withdraw 1 DOT from relay
    { "type": "WithdrawAsset", "value": [
      {
        "id": { "parents": 1, "interior": { "type": "Here" } },
        "fun": { "type": "Fungible", "value": 1000000000000 }
      }
    ]},

    { "type": "BuyExecution", "value": {
      "fees": {
        "id": { "parents": 1, "interior": { "type": "Here" } },
        "fun": { "type": "Fungible", "value": 1000000000000 }
      },
      "weight_limit": { "type": "Unlimited" }
    }},

    { "type": "DepositAsset", "value": {
      "assets": { "type": "Wild", "value": { "type": "All" } },
      "beneficiary": {
        "parents": 0,
        "interior": {
          "type": "X1",
          "value": [{
            "type": "AccountId32",
            "value": { "id": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d" }
          }]
        }
      }
    }}
  ]
}
Medium: reserve transfer 
{
  "type": "V5",
  "value": [
    { "type": "SetFeesMode", "value": { "jit_withdraw": true } },

    { "type": "WithdrawAsset", "value": [
      {
        "id": {
          "parents": 0,
          "interior": { "type": "X2", "value": [
            { "type": "PalletInstance", "value": 50 },
            { "type": "GeneralIndex", "value": 3 }
          ]}
        },
        "fun": { "type": "Fungible", "value": 100000000000 }
      }
    ]},

    { "type": "InitiateTransfer", "value": {
      "destination": {
        "parents": 1,
        "interior": { "type": "X1", "value": [
          { "type": "Parachain", "value": 1004 }
        ]}
      },
      "remote_fees": { "type": "ReserveDeposit", "value": {
        "type": "Definite", "value": [
          {
            "id": {
              "parents": 0,
              "interior": { "type": "X2", "value": [
                { "type": "PalletInstance", "value": 50 },
                { "type": "GeneralIndex", "value": 3 }
              ]}
            },
            "fun": { "type": "Fungible", "value": 1000000000 }
          }
        ]
      }},
      "preserve_origin": false,
      "assets": [
        { "type": "ReserveDeposit", "value": { "type": "Wild", "value": { "type": "All" } } }
      ],
      "remote_xcm": [
        { "type": "SetFeesMode", "value": { "jit_withdraw": true } },
        { "type": "DepositAsset", "value": {
          "assets": { "type": "Wild", "value": { "type": "All" } },
          "beneficiary": {
            "parents": 0,
            "interior": { "type": "X1", "value": [
              { "type": "AccountId32", "value": {
                "id": "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"
              }}
            ]}
          }
        }}
      ]
    }}
  ]
}

Pros:

  • Zero learning curve -- it is the JSON the CLI already accepts, plus comments
  • Near-zero parser work: strip comments + trailing commas, then JSON.parse()
  • Full compatibility with existing normalizeValue() pipeline
  • Existing JSONC tooling, syntax highlighting
  • Can be generated/consumed by any JSON library

Cons:

  • Still very verbose -- {type, value} wrappers dominate visual space
  • No variables/reuse
  • No numeric underscores
  • Locations are just as painful as inline CLI args
  • Barely an improvement over the current inline approach (just adds comments + formatting)

Parse pipeline: Strip comments -> JSON.parse() -> feed directly into normalizeValue()

Format 4: JSON5/JSONC + location shorthand

JSON5 with location strings expanded. Same {type, value} for enums but compact paths for locations.

Small: simple teleport 
{
  "type": "V5",
  "value": [
    { "type": "WithdrawAsset", "value": [
      { "id": "..", "fun": { "type": "Fungible", "value": 1000000000000 } }
    ]},

    { "type": "BuyExecution", "value": {
      "fees": { "id": "..", "fun": { "type": "Fungible", "value": 1000000000000 } },
      "weight_limit": { "type": "Unlimited" }
    }},

    { "type": "DepositAsset", "value": {
      "assets": { "type": "Wild", "value": { "type": "All" } },
      "beneficiary": "./AccountId32(0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d)"
    }}
  ]
}
Medium: reserve transfer 
{
  "type": "V5",
  "value": [
    { "type": "SetFeesMode", "value": { "jit_withdraw": true } },

    { "type": "WithdrawAsset", "value": [
      {
        "id": "./PalletInstance(50)/GeneralIndex(3)",
        "fun": { "type": "Fungible", "value": 100000000000 }
      }
    ]},

    { "type": "InitiateTransfer", "value": {
      "destination": "../Parachain(1004)",
      "remote_fees": { "type": "ReserveDeposit", "value": {
        "type": "Definite", "value": [
          {
            "id": "./PalletInstance(50)/GeneralIndex(3)",
            "fun": { "type": "Fungible", "value": 1000000000 }
          }
        ]
      }},
      "preserve_origin": false,
      "assets": [
        { "type": "ReserveDeposit", "value": { "type": "Wild", "value": { "type": "All" } } }
      ],
      "remote_xcm": [
        { "type": "SetFeesMode", "value": { "jit_withdraw": true } },
        { "type": "DepositAsset", "value": {
          "assets": { "type": "Wild", "value": { "type": "All" } },
          "beneficiary": "./AccountId32(0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d)"
        }}
      ]
    }}
  ]
}

Pros:

  • Location shorthand provides significant compression
  • Still JSON -- minimal learning curve
  • Parser: JSON5 parse + metadata-aware tree walk to expand location strings

Cons:

  • Enums still verbose ({type, value} everywhere)
  • No variables
  • No numeric underscores
  • Half-measure: locations are nice, everything else is still painful

Parse pipeline: JSON5.parse() -> walk tree, expand location strings where metadata says the field is a Location type -> feed into normalizeValue()

Evaluation matrix

Criterion Pure YAML YAML + sugar Pure JSON5 JSON5 + locations
Location readability Poor Excellent Poor Good
Enum readability Good Very good Poor Poor
Deep nesting OK Good OK OK
Variables/reuse YAML anchors vars/$ None None
Comments Yes Yes Yes Yes
Numeric underscores No Yes (strings) No No
Parser complexity Medium Medium-high Very low Low
Learning curve Low Low-medium Very low Very low
Editor tooling Excellent Good Excellent Excellent
Error messages Good (lib) Good + custom Good (lib) Good (lib)
Metadata awareness needed Yes (enum maps) Yes (enum maps + shorthand) No Yes (location fields)
Lines of code (est.) ~200 ~400 ~30 ~150

Integration sketch

CLI surface
# New flag on dot tx
dot tx --file transfer.xcm.yaml --from alice --chain assethub-preview

# Or new command (future, richer features)
dot xcm run transfer.xcm.yaml --from alice --chain assethub-preview
Parse pipeline (any format)
file content
  -> format-specific parser (YAML/JSON5)
  -> variable substitution (if supported)
  -> shorthand expansion (if supported)
  -> convert to {type, value} JSON
  -> existing normalizeValue() in tx.ts
  -> polkadot-api SCALE encoder
  -> submit via dot tx machinery
Key integration point

The XCM file parser's output must match what parseTypedArg() in src/commands/tx.ts currently produces from inline JSON args. Specifically, the normalizeValue() function (line 479) handles:

  • {type, value} discriminated unions
  • Single-element array unwrapping (XCM X1 junctions)
  • Recursive struct/array/tuple normalization

Open questions

  1. Should variables support expressions (e.g. arithmetic on amounts) or just simple substitution?
  2. Should the file include the pallet call (e.g. PolkadotXcm.execute) or just the XCM program body?
  3. How to handle the weight_limit / max_weight parameter that PolkadotXcm.execute requires alongside the XCM message?
  4. Should multiple XCM programs be composable in one file (e.g. for batch operations)?
  5. For YAML format: how to handle YAML's implicit type coercion (hex strings, booleans)?

References

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the existing dot tx --file entry point and the normalizeValue() pipeline, using scripts/setup-pusd.sh as the real-world example. Resolve the file format and scope choices in this RFC, then define parsing and validation behavior for .xcm.yaml or .xcm.json; done means an XCM program can be loaded from a file instead of inline JSON.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.