2021-06-01
Today I’d like to illustrate one important design aspect of TAO.
It is the problem of how to interleave structure and data that goes together with it within a minimal syntax.
This is related to escaping and more generally in-band signalling.
The problems that appear in most designs are the leaning toothpick syndrome and delimiter collision.
I have explored many solutions to these problems, including inventing elaborate mechanisms that effectively allow in-band redefinition of syntax, only to conclude that they conflict with the basic design goals of simplicity and minimalism.
Thus I settled on the most practical approach that meets these goals, i.e. the syntax has a minimal number of special symbols (3) that, on average, occur relatively rarely. One of the symbols (`) provides a generalized escaping mechanism. Next to that it is a separator and an extension mechanism.
This seems to be a pragmatic solution in most contexts, but it is not ideal. So I continue to explore the space in search for variants of TAO that will fit in the remaining contexts.
These variants could use control characters (e.g.) for the special symbols, virtually eliminating the aforementioned problems.