NTCIR System

TeX Toknizer

The TeX Tokenizer is based on the standard java tokenizer and transforms a TeX string to a multiset of tokens discoverd in the text. It is also used to transofrm TeXQuers as e.g. \sum\qvar{p}_{n}\qvar{a}_{n} to a so called TeX-Filter. A similar function transform Presentation MathML to a multiset of Presentation MathML tokens. (see PMML-Tokenizer) import java.util.StringTokenizer;