Parser Project Manifesto

Purpose

• Get the parser & lexer to a state where they can be worked on again. [Done]
• Fix niggling bugs in the parser and lexer. (Parser has been unchanged for some time despite many niggling bugs.) [Done]
• I’m afraid of the current parser and lexer. I don’t think I can make changes in it. [Done]
• …

Principles

• Not controversial
• Experimentation over debates. Lets see what changes feel like in practise rather than endless debating and inaction.
• …

Vision

• maintained [Done]
• evolving like the rest of Io [Done]
• is understood
• helpful error reporting
• tested
• clean code

Brainstorm

1. In what (wacky) ways might the Io parser stack evolve?
   • dynamic precedence [Done]
   • new operators at runtime [Done]
   • faster implementations
   • support for “C operator precedence and ()” [Done]
   • plug-in Io implementations of different parser stages at runtime [Done]
   • stream based parsing; start message execution before fully parsing the source
   • new line escaping; \ at the end of lines [Done]
   • generating the parser’s source from an Io description of the parser
   • ignore trailing empty comma

     foo(a,b,c,)
     
     if(c,
        x
     ,
        
     )

   • overhaul of “syntax error” reporting / making errors helpful rather than confusing
   • have all operators that end in = be transformed as setSlot is [Done; by allowing custom = style operators]
   • a generic lexer; sort of a little BNF vm
   • Adding the ability for messages to distinguish between "foo" and "foo()".
   • give quag chocolate [Not done!]
2. Layers

   “Lexer”

   creates a stream of tokens

   “Parser”

   converts token stream into basic messages

   “Op transform”

   manipulates a basic message tree to handle the operators

   • Running the op transformer twice should not give different results than running it once.

3. Documentation
   • Document the parser implementation
   • Document each layer’s role
   • Document the resulting syntax (There appears to be a a few obscure syntax rules. Like *** being an operator.)
4. I don’t see major problems with Io’s syntax, but I do think it is unhealthy for the parser to stagnate, particularly when bugs go unfixed for long periods of time because they are perceived as requiring to much effort to solve.
   • “:” is parsed as “:=” [Done]
   • “a :=(b) c” issue [Done]
   • Syntax errors aren't helpful for tracking down the cause.

Bugs

Lexer

• message(1..2) → 1..2 instead of 1 ..(2)
• message("abcd"..2) → "abcd" ..2 instead of "abcd" ..(2)

New Parser

New Op Shuffler

• message(x +-+ y) → x +-+ y instead of x +-+(y)

• Io> message(1 - 2 == 3)
   ==> 1 -(2) ==(3)
   Io> message(1 - 2 ==(3))
   ==> 1 -(2 ==(3))

Old Parser

Frustrations

Sub-Projects

• Understood Parser
• Tested Parser
• Helpful Errors From Parser
• Nice Implementation of Parser
• Evolving Parser

Next Actions

Small actions which can be down now (not waiting on anything) that would advance the project.

• Resolve the current bugs in the new parser and op shuffler (2nd and 3rd layers) (Nice Implementation sub-project) [Done]
• Make notes on the implementation of the new parser and op shuffler (Understood sub-project)
• Build a simple parsing test framework (Tested sub-project)
• Identify and record some tests to be implemented (Tested sub-project)
• Flesh out each of the ‘wacky’ ideas (Evolving Parser sub-project)
  • purposes and principles of each idea
  • what is the vision, ‘the dream’, that the idea would provide
  • don’t be negative about the idea
• Identify and record some cases where the parser’s syntax errors could be improved (Helpful Errors sub-project)