gren-lang / gren-lang/compiler-common

Expand the recording of the original string for some literals

Open
#34 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
No language data
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Previously we recorded the author's original string for floats, as gren-format had no idea of how the original author formatted them.
It turns out we probably need to do that for more types of literals.

I discovered this for string literals that contain unicode escapes; the parser coverts to those unicode escapes to actual utf-8 sequences, and gren-format doesn't know the original representation of them. This happens for hex integers too, although, this somehow feels less important of a concern.

Given this input:

module Repro exposing (..)


emoji =
    "\u{1F600}"


letterA =
    '\u{0041}'


carriageReturn =
    "\u{000D}"


hexLower =
    0xff


hexPadded =
    0x00FF


floatKept =
    1.5e+3

If we look at the output from gren-format --pre-ast, we see how "emoji" and "hexPadded" have been interpreted in a lossy way, such that gren-format can't re-write them in the same way the author originally wrote them.

 % ./gren-format/gren-format.sh --pre-ast foo.gren | grep -E "(value)"
      "value": "Repro"
    "values": [
        "value": {
          "value": {
              "value": "emoji"
              "value": {
                "value": "😀"
        "value": {
          "value": {
              "value": "letterA"
              "value": {
                "value": "A"
        "value": {
          "value": {
              "value": "carriageReturn"
              "value": {
                "value": "\r"
        "value": {
          "value": {
              "value": "hexLower"
              "value": {
                "value": {
                  "value": 255
        "value": {
          "value": {
              "value": "hexPadded"
              "value": {
                "value": {
                  "value": 255
        "value": {
          "value": {
              "value": "floatKept"
              "value": {
                "value": {
                  "value": 1500

I could work around the emoji case, by looking at the span of the string literal, and deducing whether it was written as a unicode escape sequence or not, but that's hacky.

It seems that we ought to capture the original string for Integers, Hex, CharLiteral, and StringLiteral too, and maybe PChr and PStr too?
That's a lot of strings, so maybe a better way ought to be invented?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the parser and the gren-format --pre-ast output for the literal kinds named in the issue: Integer, Hex, CharLiteral, StringLiteral, PChr, and PStr. Compare the sample input with its emitted representation and determine how original spellings can be retained without losing the decoded values; done means the formatter can preserve relevant escape, hexadecimal, and numeric forms.

Written by the indexing model from the issue text.

Assessment

Domain
compilers
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.