Finite State Machines: A Tour of Python's Grammar Parser
Details
Python's parser/parser generator is responsible for incorporating the rules defined in Python's grammar into a valid block of code.
We'll take an academic look at non-deterministic and deterministic finite automata. Next we'll look at how Python's team implemented this in python (see Parser/pgen.c) by walking through the code.
Hopefully you'll leave with the skills required to write your very own grammar parser.
I'm going to try and order burritos to go with our beer this time. Pizza is getting a little old, right?
UPDATE:
Hey guys, I've been doing a bit of research and have some homework reading for you all! If you want to brush up on parsing in Python you can get access to the low-level interface/parse tree with the parser module here:
https://docs.python.org/3.7/library/parser.html
Although they recommend using the ast module for interacting with the tree.
We'll just mention in passing the lexical analyzer; which feeds tokens into the parser. You can find documentation on it here:
https://docs.python.org/3.7/reference/lexical_analysis.html
Here's a pretty good run-through on finite state machines: https://en.wikipedia.org/wiki/Finite-state_machine
There was a kind of interesting conversation going on about peak memory usage by the parser lately. You can find the thread here:
https://mail.python.org/pipermail/python-dev/2016-August/145803.html
Guido shot it down with a comment to the affect of "sometimes things are too hard to be worth doing". I can't remember the exact words but I'll dig them up. I think I tweeted it recently.
I'm having a bit of trouble digging up cool stuff from the dev mailing list but I'll try to scrape together some more historical context soon.
I would also recommend checking out the "dragon book" on compilers for it's section on Finite State Automata:
which is the basis for Python's algorithm.
See you all in a week!
Finite State Machines: A Tour of Python's Grammar Parser