Main Page | Namespace List | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members

tokenizer.h File Reference

#include <set>
#include <map>
#include <vector>
#include "language.h"
#include "regexp.h"

Include dependency graph for tokenizer.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

class  tokenizer
 Class tokenizer implements a token splitter, which converts a string into a sequence of word objects, according to a set of tokenization rules read from aconfiguration file. More...


Generated on Wed Apr 26 12:57:19 2006 for FreeLing by  doxygen 1.4.4