Meta design: Use JSON equivalent representation with source range as secrets

Alternative to https://github.com/PistonDevelopers/dyon/issues/182

The meta converter language does not support repeated conversions, which is required if you need some complex rewrite logic (e.g. proof assistants). This idea is an attempt to separate the parsing step and use a more general system for tree transformation.

Piston-Meta already uses a JSON converter to debug meta data. Since Dyon is very similar to Javascript, perhaps converting meta data to a Dyon data structure directly would make it easier to work with it? This structure is easy to debug, and because Dyon is dynamically typed it is easy to transform the structure gradually, without using too much memory.

Instead of using a meta converter language, one could generate the JSON equivalent representation of the meta data. The source range could be stored as secrets, thereby keeping the trace back to the source even under complicated transformations.

Example

Similar example to https://github.com/PistonDevelopers/dyon/issues/182

You want to parse a custom format looking like this:

assets/test_data.txt

0 1 2
3 4 5
6 7 8

So, you write a meta syntax file:

assets/test_meta.txt

1 pos = [.w? .$_:"x" .w! .$_:"y" .w! .$_:"z" .w?]
0 doc = .l(pos:"pos")

In Dyon, you call load__meta_file:

data := unwrap(load(meta: "assets/test_meta.txt", file: "assets/test_data.txt"))
println(data)

This generates the Dyon data:

{
  pos: {x: 0, y: 1, z: 2},
  pos: {x: 3, y: 4, z: 5},
  pos: {x: 6, y: 7, z: 8}
}

To get the source range in characters:

println(sec(data[0].x)) // prints `some([(0, 1)])`

Unsolved problem: How to repeat meta data

In the example above, I assumed the following problem was solved:

The JSON representation in Piston-Meta is not correct for all input, since it replaces the same keys with new values when a sub rule is repeated within same parent node.

To solve this problem, it might require a change to Piston-Meta. It need to pass on information about data generated from repetition rules. Repetition rules are .l (lines), .r (repeat), .s (separated by).

One challenge is that recursion can also be repetitive when calling a sub rule without generating a new node. Perhaps this would give ideas to solve problems with left recursion?

Add support for secrets to str

Currently, Dyon only supports secrets for bool and f64. This is used in the mathematical loops.

To trace back text read from an input file, it requires support for str secrets in Dyon.

Add sec intrinsic to reveal secret of bool, f64 and str

Get the secret of bool, f64 and str. Returns none() if there is no secret.

fn sec(any) -> opt[[any]] { ... }

This is required because the existing functions are type specific and does not work in all cases for parsing. why checks for true and where checks for non-nan. These are designed to be used with mathematical loops and detect logical bugs at run-time. Need the more general sec to do this on all parsed data.

Rewriting and transformation

The following strategies can be used to transform the Dyon structure into a target structure:

• Visitor pattern with closures to walk over the tree
• Rewrite nodes using depth-first
• Converter language could generate Dyon => Dyon

Add if <pat> ?= <rexpr> { ... } for pattern matching

An idea is to add some sugar for pattern matching to the Dyon language, in order to make tree transformations easier to deal with:

x := [1, 2]
if [a, b] ?= x {
    println(a)
    println(b)
}

For functions that returns res, one could use <pat> ?= <rexpr>:

fn foo() -> res {
    x := [1, 2]
    [a, b] ?= x
    ...
}

No implementation files or tests are named. Start with the load__meta_file entry point, the existing secret handling for bool and f64, and Piston-Meta's JSON conversion of repeated keys; the scope is complete only after one approach is chosen for repetition, string secrets, sec, and transformation support.