Uses of Class
opennlp.tools.tokenize.TokenSample

Packages that use TokenSample

opennlp.tools.cmdline.tokenizer
opennlp.tools.formats	Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang.en

Uses of TokenSample in opennlp.tools.cmdline.tokenizer

Methods in opennlp.tools.cmdline.tokenizer that return types with arguments of type TokenSample

protected ObjectStreamFactory<TokenSample>

TokenizerConverterTool.createStreamFactory(String format)

Uses of TokenSample in opennlp.tools.formats

Methods in opennlp.tools.formats that return TokenSample

TokenSample	POSToTokenSampleStream.read()
TokenSample	NameToTokenSampleStream.read()

Methods in opennlp.tools.formats that return types with arguments of type TokenSample

ObjectStream<TokenSample>	POSToTokenSampleStreamFactory.create(String[] args)
ObjectStream<TokenSample>	NameToTokenSampleStreamFactory.create(String[] args)
ObjectStream<TokenSample>	ConllXTokenSampleStreamFactory.create(String[] args)

Uses of TokenSample in opennlp.tools.tokenize

Methods in opennlp.tools.tokenize that return TokenSample

static TokenSample	TokenSample.parse(String sampleString, String separatorChars)
TokenSample	TokenSampleStream.read()
TokenSample	TokenizerStream.read()

Methods in opennlp.tools.tokenize with parameters of type TokenSample

protected Iterator<opennlp.model.Event>	TokSpanEventStream.createEvents(TokenSample tokenSample) Adds training events to the event stream for each of the specified tokens.
void	TokenizerEvaluator.evaluateSample(TokenSample reference) Evaluates the given reference TokenSample object.

Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample

void	TokenizerCrossValidator.evaluate(ObjectStream<TokenSample> samples, int nFolds)
static TokenizerModel	TokenizerME.train(String languageCode, ObjectStream<TokenSample> samples, boolean useAlphaNumericOptimization) Trains a model for the TokenizerME with a default cutoff of 5 and 100 iterations.
static TokenizerModel	TokenizerME.train(String languageCode, ObjectStream<TokenSample> samples, boolean useAlphaNumericOptimization, int cutoff, int iterations) Trains a model for the TokenizerME.

Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample

TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)
Initializes the current instance.

TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)
Initializes the current instance.

WhitespaceTokenStream(ObjectStream<TokenSample> tokens)

Uses of TokenSample in opennlp.tools.tokenize.lang.en

Methods in opennlp.tools.tokenize.lang.en that return TokenSample

TokenSample

TokenSampleStream.next()