3 Commits

Author	SHA1	Message	Date
Eliot Jones	c22a79f327	more tests for array tokenization	2017-11-12 14:58:34 +00:00
Eliot Jones	158cd5f2e3	support for tokenizing arrays and nested arrays	2017-11-12 14:42:01 +00:00
Eliot Jones	00e3d06513	add some more tokens and test the core token scanner for some initial test cases, seems like the approach should be valid.	2017-11-12 01:08:09 +00:00