758 Commits

Author	SHA1	Message	Date
Eliot Jones	c22a79f327	more tests for array tokenization	2017-11-12 14:58:34 +00:00
Eliot Jones	158cd5f2e3	support for tokenizing arrays and nested arrays	2017-11-12 14:42:01 +00:00
Eliot Jones	00e3d06513	add some more tokens and test the core token scanner for some initial test cases, seems like the approach should be valid.	2017-11-12 01:08:09 +00:00
Eliot Jones	096278de50	create name tokenizer and approach for reading non variable data tokens	2017-11-10 23:58:28 +00:00
Eliot Jones	f986e16c97	rename hex tokenizer and add numeric tokenizer	2017-11-10 21:13:45 +00:00
Eliot Jones	8dbeb4b822	quick initial draft of the hex parser. very inefficient	2017-11-09 23:26:21 +00:00
Eliot Jones	83cc1a6bf1	fixes for octal in strings and tests for string and hex tokenizing	2017-11-09 22:52:48 +00:00
Eliot Jones	fd53d91c28	move code into correct directory structure	2017-11-09 19:22:52 +00:00