Class TokenizerModel

    • Constructor Detail

      • TokenizerModel

        public TokenizerModel​(MaxentModel tokenizerModel,
                              java.util.Map<java.lang.String,​java.lang.String> manifestInfoEntries,
                              TokenizerFactory tokenizerFactory)
        Initializes the current instance.
        Parameters:
        tokenizerModel - the model
        manifestInfoEntries - the manifest
        tokenizerFactory - the factory
      • TokenizerModel

        public TokenizerModel​(java.io.InputStream in)
                       throws java.io.IOException
        Initializes the current instance.
        Parameters:
        in - the Input Stream to load the model from
        Throws:
        java.io.IOException - if reading from the stream fails in anyway
        InvalidFormatException - if the stream doesn't have the expected format
      • TokenizerModel

        public TokenizerModel​(java.io.File modelFile)
                       throws java.io.IOException
        Initializes the current instance.
        Parameters:
        modelFile - the file containing the tokenizer model
        Throws:
        java.io.IOException - if reading from the stream fails in anyway
      • TokenizerModel

        public TokenizerModel​(java.nio.file.Path modelPath)
                       throws java.io.IOException
        Throws:
        java.io.IOException
      • TokenizerModel

        public TokenizerModel​(java.net.URL modelURL)
                       throws java.io.IOException
        Initializes the current instance.
        Parameters:
        modelURL - the URL pointing to the tokenizer model
        Throws:
        java.io.IOException - if reading from the stream fails in anyway
    • Method Detail

      • getAbbreviations

        public Dictionary getAbbreviations()
      • useAlphaNumericOptimization

        public boolean useAlphaNumericOptimization()