Lexical Processing

Transforming a text into a sequence of atomic entities for further analysis.

Examples

• Tokenize a text based on characters
• Tokenize a text based on words
• Tokenize a text based on sentences
• Tokenize a text based on chapters