opennlp.tools.tokenize
Interface Detokenizer

All Known Implementing Classes:

DictionaryDetokenizer

public interface Detokenizer

A Detokenizer merges tokens back to their untokenized representation.

Nested Class Summary

static class

Detokenizer.DetokenizationOperation This enum contains an operation for every token to merge the tokens together to their detokenized form.

Method Summary

Detokenizer.DetokenizationOperation[]

detokenize(String[] tokens) Detokenize the input tokens.

Method Detail

detokenize

Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Detokenize the input tokens.

Parameters:

tokens - the tokens to detokenize.

Returns:

the merge operations to detokenize the input tokens.