racket / racket/syntax-color

scribble lexer returns an empty token

Open
#14 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Racket
Stars
8
Forks
16
PR merge metrics
No merged PRs in 30d

Description

This was found by random testing.

Here's the (non-random) test case that illustrates the bug. The function try should never raise an error (it is a simplification of the checks done by the option in the contract). The string big-s is the one that drdr found and it shrinks to "^|{|\\".

#lang racket
(require syntax-color/scribble-lexer)

(define (try s)
  (define 3ary-lexer (make-scribble-inside-lexer #:command-char #\^))
  (define in (open-input-string s))
  (port-count-lines! in)
  (let loop ([mode #f][offset 0])
    (define-values (txt type paren start end backup new-mode)
      (3ary-lexer in offset mode))
    (cond
      [(equal? type 'eof) #t]
      [else
       (unless (< start end)
         (error 'whoops! "start ~s end ~s" start end))
       (loop new-mode end)])))

;; returns #t if the test case failed, #f if it didn't fail
;; so `s` is helpful when this returns #t
(define (try/bool s)
  (with-handlers ([(λ (x) (and (exn:fail? x)
                               (regexp-match #rx"whoops" (exn-message x))))
                   (λ (x) #t)])
    (try s)
    #f))

(define big-s
  "\"\u0099\"\u009E\u0095Ýäλ|Á5|\u008D)è_)4\u0097(¬{\u009D })Ó|@^|{E cTuÐÆ}\u009E|\\|\u0017)\u0099j\u009A")

(define (get-candidates s)
  (append
   (for/list ([i (in-range (string-length s))])
     (string-append (substring s 0 i)
                    (substring s (+ i 1) (string-length s))))
   (for/list ([i (in-range (string-length s))]
              #:unless (equal? (string-ref s i) #\a))
     (string-append (substring s 0 i)
                    "a"
                    (substring s (+ i 1) (string-length s))))))

(define (shrink s)
  (let loop ([s s])
    (cond
      [(try/bool s)
       (define candidates (get-candidates s))
       (cond
         [(null? candidates) s]
         [else
          (or (for/or ([candidate (in-list candidates)])
                (loop candidate))
              s)])]
      [else #f])))

(shrink big-s)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with syntax-color/scribble-lexer and the make-scribble-inside-lexer entry point described in the issue. Run the provided try case with the minimized string "^|{\" and inspect the returned token positions; done means the lexer no longer produces an empty token that makes start fail to be less than end.

Written by the indexing model from the issue text.

Assessment

Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.