opennlp.tools.tokenize
Class TokenSample

java.lang.Object
  opennlp.tools.tokenize.TokenSample

public class TokenSample
extends Object

A TokenSample is text with token spans.

Field Summary

static String

DEFAULT_SEPARATOR_CHARS

Constructor Summary

TokenSample(Detokenizer detokenizer, String[] tokens)

TokenSample(String text, Span[] tokenSpans)
Initializes the current instance.

Method Summary

String	getText() Retrieves the text.
Span[]	getTokenSpans() Retrieves the token spans.
static TokenSample	parse(String sampleString, String separatorChars)
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final String DEFAULT_SEPARATOR_CHARS

See Also:

Constant Field Values

Constructor Detail

TokenSample

public TokenSample(String text,
                   Span[] tokenSpans)

Initializes the current instance.

Parameters:

text - the text which contains the tokens.

tokenSpans - the spans which mark the begin and end of the tokens.

TokenSample

public TokenSample(Detokenizer detokenizer,
                   String[] tokens)

Method Detail

getText

public String getText()

Retrieves the text.

getTokenSpans

public Span[] getTokenSpans()

Retrieves the token spans.

toString

public String toString()

Overrides:

toString in class Object

parse

public static TokenSample parse(String sampleString,
                                String separatorChars)