Writing A Compiler In Go

t's the sequel to … a programming language

Writing A Compiler In Go is the sequel to Writing An Interpreter In Go. It starts right where the first one stopped, with a fully-working, fully-tested Monkey interpreter in hand, connecting both books seamlessly, ready to build a compiler and a virtual machine for Monkey.

In this book, we use the codebase (included in the book!) from the first part and extend it. We take the lexer, the parser, the AST, the REPL and the object system and use them to build a new, faster implementation of Monkey, right next to the tree-walking evaluator we built in the first book.

The approach is unchanged, too. Working, tested code is the focus, we build everything from scratch, do baby steps, write tests firsts, use no 3rd-party-libraries and see and understand how all the pieces fit together.

It's a continuation in prose and in code.

Do you need to read the first part before this one? If you're okay with treating the code from the first book as a black box, then no. But that's not what these books are about; they're about opening up black boxes, looking inside and shining a light. You'll have the best understanding of where we're going in this book, if you know where we started.

Learn how to write a compiler and a virtual machine

Our main goal in in this book is to evolve Monkey. We change its architecture and turn it into a bytecode compiler and virtual machine.

We'll take the lexer, the parser, the AST and the object system we wrote in the first book and use them to build our own Monkey compiler and virtual machine … from scratch! We'll build them side-by-side so that we'll always have a running system we can steadily evolve.

What we end up with is not only much closer to the programming languages we use every day, giving us a better understanding of how they work, but also 3x faster. And that's without explicitly aiming for performance.

Here's what we'll do:

• We define our own bytecode instructions, specifying their operands and their encoding. Along the way, we also build a mini-disassembler for them.
• We write a compiler that takes in a Monkey AST and turns it into bytecode by emitting instructions
• At the same time we build a stack-based virtual machine that executes the bytecode in its main loop

We'll learn a lot about computers, how they work, what machine code and opcodes are, what the stack is and how to work with stack pointers and frame pointers, what it means to define a calling convention, and much more.

We also

• build a symbol table and a constant pool
• do stack arithmetic
• generate jump instructions
• build frames into our VM to execute functions with local bindings and arguments!
• add built-in functions to the VM
• get real closures working in the virtual machine and learn why closure-compilation is so tricky