IppClub / IppClub/YueScript

[Feature request] Additional parsing functions for the 'yue'-module

Open
#196 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Lua
Stars
613
Forks
45
PR merge metrics
No merged PRs in 30d

Description

It would be nice to have the following functions be added to the yue module:

  • yue.to_tokens(code: string): string[]: Turns a chunk of YueScript code into a list of tokens, so local even_nums = [i for 1, 10 when (i % 2) == 0] would become something like [ "local", " ", "even_nums", " ", "=", " ", "[", "i", " ", "for", " ", "i", " ", "=", " ", "1", ",", " ", "10", " ", "when", " ", "(", "i", " ", "%", " ", "2", ")", " ", "==", " ", "0", "]" ]. This would be particularly useful for macros because the input might not make sense when parsed into a YueScript AST.
  • yue.to_cst(code: string): yue.CstNode: A CST (concrete syntax tree) is similar to an AST, but unlike an AST, it contains all information required to reconstruct the exact source code that it was generated from. An AST would discard information like whitespace and comments, while a CST would keep them. A CST should also ideally be a strict superset of an AST, so that a node visitor for an AST can also operate on the same node when used with a CST.

As a side note: It may be worth considering replacing the current macro semantics with something more like Rust's approach of taking a single list of tokens and returning a list of tokens, instead of working with source code. Doing it this way is easier to use as well as more reliable than trying to monkey-patch some strings with Lua patterns / regular expressions.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the yue module, its current parser, and the macro entry points mentioned in the issue. Define how token lists and a CST should represent the supplied YueScript example, including whitespace and comments. Done means the module exposes both parsing functions and the CST can preserve enough information to reconstruct the original source.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.