opennlp.tools.tokenize
Class TokenizerModel

java.lang.Object
  opennlp.tools.util.model.BaseModel
      opennlp.tools.tokenize.TokenizerModel

public final class TokenizerModel
extends BaseModel

The TokenizerModel is the model used by a learnable Tokenizer.

See Also:

TokenizerME

Field Summary

Fields inherited from class opennlp.tools.util.model.BaseModel

artifactMap, MANIFEST_ENTRY, TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

TokenizerModel(InputStream in)
Initializes the current instance.

TokenizerModel(String language, opennlp.model.AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization)
Initializes the current instance.

TokenizerModel(String language, opennlp.model.AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries)
Initializes the current instance.

Method Summary

opennlp.model.AbstractModel	getMaxentModel()
static void	main(String[] args)
boolean	useAlphaNumericOptimization()
protected void	validateArtifactMap() Validates the parsed artifacts.

Methods inherited from class opennlp.tools.util.model.BaseModel

createArtifactSerializers, getArtifactSerializer, getLanguage, getManifestProperty, getVersion, serialize, setManifestProperty

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerModel

public TokenizerModel(String language,
                      opennlp.model.AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization,
                      Map<String,String> manifestInfoEntries)

Initializes the current instance.

Parameters:

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(String language,
                      opennlp.model.AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization)

Initializes the current instance.

Parameters:

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(InputStream in)
               throws IOException,
                      InvalidFormatException

Initializes the current instance.

Parameters:

in -

Throws:

IOException

InvalidFormatException

Method Detail

validateArtifactMap

protected void validateArtifactMap()
                            throws InvalidFormatException

Description copied from class: BaseModel

Validates the parsed artifacts. If something is not valid subclasses should throw an InvalidFormatException. Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.

Overrides:

validateArtifactMap in class BaseModel

Throws:

InvalidFormatException

getMaxentModel

public opennlp.model.AbstractModel getMaxentModel()

useAlphaNumericOptimization

public boolean useAlphaNumericOptimization()

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException