In Chapter 1.3 of [Ball’s Writing an Interpreter in Go][1], we encounter one design decision of his Monkey programming language. Here, the lexer has a NextToken() method that looks like this: func (l *Lexer) NextToken() token.Token { var tok token.Token switch l.ch { // [...] default: if isLetter(l.ch) { tok.Literal = l.readIdentifier() return tok } else { tok = newToken(token.ILLEGAL, l.ch) } } // [...] } This means the lexer itself does not do backtracking.

A lexer takes the source code, a sequence of characters, and group them into tokens. e.g., it makes the first decision on how to process the strings 100-10, -100-10, and -100--100 into groups. I’m going to call this grouping “tokenization” even though I may be misusing the term. Tokenizing source code is hard. How should -100--100 be tokenized? Should it be a literal -100 followed by the minus token, followed by another -100?

Writing an Interpreter In Go by Thorsten Ball will be my personal introduction to writing an interpreter. I’ve never taken a comp sci class before, so I know nothing about compilers. On a lark, I decided to explore this area now, nearly 20 years after I started to learn computer programming. If you are interested in this book as well, you might might the AST Explorer a useful companion. I was told as some point in the past, that compilation can be broken down into four stages:

It happened. At the recommendation of https://twitter.com/dgryski, I bought Writing an Interpreter In Go. This will be my next hobby project. It’ll be interesting to see if I ever finish it.