C[omp]ute

From: "andrew cooke" <andrew@...>

Date: Sun, 19 Apr 2009 08:56:06 -0400 (CLT)

Spent yesterday travelling and, in an attempt to get work out of my head,
hacked on LEPL's tokenizer support.

I never completed a comment discussing the syntax, but the following shows
how it integrates nicely into what is already there.  Note, for example,
the use of Float() inside Token().  This uses the "compile to regexp"
support I added earlier to convert the Float() matcher to a regexp that's
then used by the tokenizer.

Also, note how handling of spaces becomes much simpler with a lexer
because it's no longer explicit in the grammar (well, only in the token
defs).

Another feature, hardly exploited below, is that you can match inside a
token.  For example symbol('(') requires that (i) a symbol token exist and
(ii) that it matches '(' (whether the match requires the entire token or
not is currently undefined - I think it would be best if it does by
default).

So, here's the test code and output:

  class Call(Node): pass
  class Term(Node): pass
  class Factor(Node): pass
  class Expression(Node): pass

  value  = Token(Float())                       > 'number'
  name   = Token('[a-z]+')
  symbol = Token('[^a-zA-Z0-9\\. ]')

  expr    = Delayed()
  open    = symbol('(')
  close   = symbol(')')
  funcn   = name                                > 'name'
  call    = funcn & open & expr & close         > Call
  term    = call | value | open & expr & close  > Term
  muldiv  = symbol(Any('*/'))                   > 'operator'
  factor  = term & (muldiv & term)[:]           > Factor
  addsub  = symbol(Any('+-'))                   > 'operator'
  expr   += factor & (addsub & factor)[:]       > Expression
  line    = expr & Eos()

  parser = line.null_parser(
	      Configuration(rewriters=[lexer_rewriter]))
  parser('1 + 2*sin(3+ 4)')[0]

Expression
 +- Factor
 |   `- Term
 |       `- number '1'
 +- operator '+'
 `- Factor
     +- Term
     |   `- number '2'
     +- operator '*'
     `- Term
         `- Call
             +- name 'sin'
             +- '('
             +- Expression
             |   +- Factor
             |   |   `- Term
             |   |       `- number '3'
             |   +- operator '+'
             |   `- Factor
             |       `- Term
             |           `- number '4'
             `- ')'

Andrew