You might not know it but the most important puzzle piece for all my recent ESP8266 projects is a thing called a streaming parser. Keep reading if you want to know what that is, how it works and why it is so important for my ESP8266 projects.
What is a streaming parser?
What is a parser anyway? You are most certainly using parsers every day. A parser is a piece of code that analyses an input (text, document) by reading in its content based. To do that the parser has knowledge about the structure of the text, sometimes called a syntax. The syntax is like the grammar of your natural language. The web browser you are using to read this text uses an HTML parser to understand the tags that are downloaded from my webserver and then put into a visualization with formatted text, pictures and links.
So now that we roughly understand what a parser is the next question would be what is a streaming parser? With a modern smart phone or desktop computer we often don’t need streaming parsers anymore, we use document object model parsers (DOM) parsers instead. A DOM parser creates a tree like structure of the document it parses, keeps this structure in memory and makes it available for the code that does something meaningful with it. DOM parsers are very easy to use, fast and convenient. But this convenience comes at the price of memory requirements. The DOM parser needs a lot of memory, since it keeps the whole document in memory until it is no longer used. If you have a lot of RAM and your documents are not that big this is perfectly fine. But if the documents are big compared to the available (heap) memory you might run into a serious problem.
Imagine the parser to be something like a water meter and we are comparing now two different types of meters. A water meter which works like a DOM parser needs a bucket and measures the amount of water by filling the bucket and then measuring the weight of the water in the bucket. If there is a lot of water then the bucket must be big. A water meter which works like a streaming parser measures the water while it flows through and doesn’t care what happens to the water afterwards. The bucket in this analogy is the heap or working memory of your micro controller, the water is the stream of bits and bytes that you receive, either from the file system or from a remote server. And the parser does not just measure the amount of bits and bytes but also tries to understand the content. The streaming parser doesn’t care how big the document (or the amount of water) is, it just takes out what it needs from the stream. Streaming parsers are also referred to as event based parsers since they react to certain events in the data stream. DOM parser are referred to as a tree based parser since they build a full representation of the document in the tree-like structure. In an HTML tree the <html> element would be the root of the tree, the <body> tag a fork in that tree.
The grammar
The following image describes the grammar of a valid JSON object in a very concise way. It means that a JSON object knows three basic types:
- object
- array and
- value
Objects always start and end with curly brackets. They can be either empty (line to the top) or contain string/value pairs, separated by a colon. These pairs can be repeated by adding a comma between them.
Arrays start and end with square brackets. They can be either empty or contain a value. At this point we don’t know yet what a value is. Values in an array can be repeated and must be separated by a comma.
Values were already used for the two previous definitions and here lies the power of this kind of grammar. Because a value can contain a simple text, a number, an object (yes, the object we defined before!!!) an array (also defined before), booleans and a null value.
This is so powerful because we are reusing the definitions and we are nesting them within each other: an object can contain a value, a value can contain an array or an object. And finally an array can contain a value, repeatedly! Isn’t this beautiful?
The JSON Streaming Parser Library
Why would we want to use a streaming parser on the ESP8266? Embedded devices usually have very limited resources available. One scarce resource is the heap memory. Many of the REST APIs I am using in my projects provide big response objects, but we are usually just interested in a small fraction of it. As mentioned earlier, a tree based parser would load the whole document into memory and make it available once the document stream has ended. And this would just crash the ESP8266 pretty quickly, it does not have the resources to keep 200kb on the heap.
This made me port a PHP JSON parser over to C++ and make it available as a library, mostly to be used in my own projects. Let’s have a look at the header file of the JsonListener:
The methods here are callback methods which will get invoked if the respective event happens while parsing the document. Let’s start with an example. For the JSON object {“name”: “Eichhorn”} we get the following invocations:
- startDocument(): we start receiving a json document
- startObject(): the json object starts with “{“
- key(“name”): the parser detected key object which contains “name”
- value(“Eichhorn”): the parser detected a value containing “Eichhorn”
- endObject(): the object ends with “}”
- endDocument(): the stream of data ends and so does the document
I often just implement (AKA “write code”) for key and the value method. In the key method I store the value of the key parameter. Then later in the value method I check what the last key was I had seen and then I store the value in the appropriate variable. For the example from before I would do
In the stream of the object {“name”: “Eichhorn”} we will first get a call to the method key with the value “name” which we store in currentKey_. Next the parser will detect a value and call our value method with the value “Eichhorn”. The parser can now make the connection (or create a context) that after the key name the value Eichhorn should be stored in the member variable name_.
If this example was to simple then have a look here: https://github.com/squix78/esp8266-weather-station/blob/master/WundergroundClient.cpp This is the code which parses the responses from Wunderground for my WeatherStation.
Conclusion
For a document or object of the size we had in the example a streaming parser is usually an extreme overkill. It is complicated to use, requires you to write a lot of code and is memory wise probably even worse than a tree parser. It is only recommended to implement a streaming parser if you have big objects or if you just don’t know how big your object might be. In those cases a streaming parser will be a good friend, since it only requires memory for the objects you actually want to use from the whole big document. You can find my library here: https://github.com/squix78/json-streaming-parser
