Uses of Package
opennlp.tools.tokenize

Packages that use opennlp.tools.tokenize

opennlp.tools.cmdline.tokenizer
opennlp.tools.formats	Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.sentdetect	Package related to identifying sentece boundries.
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.util.featuregen	This package contains classes for generating sequence features.

Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.tokenizer

Detokenizer.DetokenizationOperation
This enum contains an operation for every token to merge the tokens together to their detokenized form.

TokenSample
A TokenSample is text with token spans.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats

Detokenizer
A Detokenizer merges tokens back to their untokenized representation.

TokenSample
A TokenSample is text with token spans.

Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect

Detokenizer
A Detokenizer merges tokens back to their untokenized representation.

Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize

DetokenizationDictionary

DetokenizationDictionary.Operation

Detokenizer
A Detokenizer merges tokens back to their untokenized representation.

Detokenizer.DetokenizationOperation
This enum contains an operation for every token to merge the tokens together to their detokenized form.

SimpleTokenizer
Performs tokenization using character classes.

TokenContextGenerator
Interface for TokenizerME context generators.

Tokenizer
The interface for tokenizers, which segment a string into its tokens.

TokenizerModel
The TokenizerModel is the model used by a learnable Tokenizer.

TokenSample
A TokenSample is text with token spans.

WhitespaceTokenizer
This tokenizer uses white spaces to tokenize the input text.

Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen

Tokenizer
The interface for tokenizers, which segment a string into its tokens.