Package org.apache.lucene.monitor
Class SuffixingNGramTokenFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.monitor.SuffixingNGramTokenFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Unwrappable<TokenStream>
final class SuffixingNGramTokenFilter extends TokenFilter
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description private java.lang.String
anyToken
private int
curCodePointCount
private int
curGramSize
private int
curPos
private int
curPosInc
private int
curPosLen
private char[]
curTermBuffer
private int
curTermLength
private KeywordAttribute
keywordAtt
private int
maxTokenLength
private OffsetAttribute
offsetAtt
private PositionIncrementAttribute
posIncAtt
private PositionLengthAttribute
posLenAtt
private CharArraySet
seenInfixes
private CharArraySet
seenSuffixes
private java.lang.String
suffix
private CharTermAttribute
termAtt
private int
tokEnd
private int
tokStart
-
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description SuffixingNGramTokenFilter(TokenStream input, java.lang.String suffix, java.lang.String wildcardToken, int maxTokenLength)
Creates SuffixingNGramTokenFilter.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
incrementToken()
Returns the next token in the stream, or null at EOS.void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end, unwrap
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
suffix
private final java.lang.String suffix
-
maxTokenLength
private final int maxTokenLength
-
anyToken
private final java.lang.String anyToken
-
curTermBuffer
private char[] curTermBuffer
-
curTermLength
private int curTermLength
-
curCodePointCount
private int curCodePointCount
-
curGramSize
private int curGramSize
-
curPos
private int curPos
-
curPosInc
private int curPosInc
-
curPosLen
private int curPosLen
-
tokStart
private int tokStart
-
tokEnd
private int tokEnd
-
termAtt
private final CharTermAttribute termAtt
-
posIncAtt
private final PositionIncrementAttribute posIncAtt
-
posLenAtt
private final PositionLengthAttribute posLenAtt
-
offsetAtt
private final OffsetAttribute offsetAtt
-
keywordAtt
private final KeywordAttribute keywordAtt
-
seenSuffixes
private final CharArraySet seenSuffixes
-
seenInfixes
private final CharArraySet seenInfixes
-
-
Constructor Detail
-
SuffixingNGramTokenFilter
public SuffixingNGramTokenFilter(TokenStream input, java.lang.String suffix, java.lang.String wildcardToken, int maxTokenLength)
Creates SuffixingNGramTokenFilter.- Parameters:
input
-TokenStream
holding the input to be tokenizedsuffix
- a string to suffix to all ngramswildcardToken
- a token to emit if the input token is longer than maxTokenLengthmaxTokenLength
- tokens longer than this will not be ngrammed
-
-
Method Detail
-
incrementToken
public final boolean incrementToken() throws java.io.IOException
Returns the next token in the stream, or null at EOS.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
reset
public void reset() throws java.io.IOException
Description copied from class:TokenFilter
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).NOTE: The default implementation chains the call to the input TokenStream, so be sure to call
super.reset()
when overriding this method.- Overrides:
reset
in classTokenFilter
- Throws:
java.io.IOException
-
-