opennlp.tools.tokenize
Class TokenSample

java.lang.Object
  extended by opennlp.tools.tokenize.TokenSample

public class TokenSample
extends Object

A TokenSample is text with token spans.


Field Summary
static String DEFAULT_SEPARATOR_CHARS
           
 
Constructor Summary
TokenSample(Detokenizer detokenizer, String[] tokens)
           
TokenSample(String text, Span[] tokenSpans)
          Initializes the current instance.
 
Method Summary
 String getText()
          Retrieves the text.
 Span[] getTokenSpans()
          Retrieves the token spans.
static TokenSample parse(String sampleString, String separatorChars)
           
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final String DEFAULT_SEPARATOR_CHARS
See Also:
Constant Field Values
Constructor Detail

TokenSample

public TokenSample(String text,
                   Span[] tokenSpans)
Initializes the current instance.

Parameters:
text - the text which contains the tokens.
tokenSpans - the spans which mark the begin and end of the tokens.

TokenSample

public TokenSample(Detokenizer detokenizer,
                   String[] tokens)
Method Detail

getText

public String getText()
Retrieves the text.


getTokenSpans

public Span[] getTokenSpans()
Retrieves the token spans.


toString

public String toString()
Overrides:
toString in class Object

parse

public static TokenSample parse(String sampleString,
                                String separatorChars)


Copyright © 2010. All Rights Reserved.