2 Commits

Author	SHA1	Message	Date
Eliot Jones	f09ef85e5a	make tokenizer classes internal and change the file header to use a scanner rather than the pdfbox type reader	2018-01-03 20:15:25 +00:00
Eliot Jones	511385a253	begin porting the character map parser over to use the new core tokenizer. add the comment tokenizer and some tests around dictionary token	2017-11-13 23:48:25 +00:00