Package org.apache.lucene.analysis.nl
Class DutchAnalyzer
- java.lang.Object
-
- org.apache.lucene.analysis.Analyzer
-
- org.apache.lucene.analysis.nl.DutchAnalyzer
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class DutchAnalyzer extends Analyzer
Analyzer
for Dutch language.Supports an external list of stopwords (words that will not be indexed at all), an external list of exclusions (word that will not be stemmed, but indexed) and an external list of word-stem pairs that overrule the algorithm (dictionary stemming). A default set of stopwords is used unless an alternative list is specified, but the exclusion list is empty by default.
- Since:
- 3.1
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
DutchAnalyzer.DefaultSetHolder
-
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DEFAULT_STOPWORD_FILE
File containing default Dutch stopwords.private CharArraySet
excltable
Contains words that should be indexed but not stemmed.private StemmerOverrideFilter.StemmerOverrideMap
stemdict
private CharArraySet
stoptable
Contains the stopwords used with the StopFilter.-
Fields inherited from class org.apache.lucene.analysis.Analyzer
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY
-
-
Constructor Summary
Constructors Constructor Description DutchAnalyzer()
Builds an analyzer with the default stop words (getDefaultStopSet()
) and a few default entries for the stem exclusion table.DutchAnalyzer(CharArraySet stopwords)
DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable)
DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable, CharArrayMap<java.lang.String> stemOverrideDict)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected Analyzer.TokenStreamComponents
createComponents(java.lang.String fieldName)
Returns a (possibly reused)TokenStream
which tokenizes all the text in the providedReader
.static CharArraySet
getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.protected TokenStream
normalize(java.lang.String fieldName, TokenStream in)
Wrap the givenTokenStream
in order to apply normalization filters.-
Methods inherited from class org.apache.lucene.analysis.Analyzer
attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReader, initReaderForNormalization, normalize, tokenStream, tokenStream
-
-
-
-
Field Detail
-
DEFAULT_STOPWORD_FILE
public static final java.lang.String DEFAULT_STOPWORD_FILE
File containing default Dutch stopwords.- See Also:
- Constant Field Values
-
stoptable
private final CharArraySet stoptable
Contains the stopwords used with the StopFilter.
-
excltable
private CharArraySet excltable
Contains words that should be indexed but not stemmed.
-
stemdict
private final StemmerOverrideFilter.StemmerOverrideMap stemdict
-
-
Constructor Detail
-
DutchAnalyzer
public DutchAnalyzer()
Builds an analyzer with the default stop words (getDefaultStopSet()
) and a few default entries for the stem exclusion table.
-
DutchAnalyzer
public DutchAnalyzer(CharArraySet stopwords)
-
DutchAnalyzer
public DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable)
-
DutchAnalyzer
public DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable, CharArrayMap<java.lang.String> stemOverrideDict)
-
-
Method Detail
-
getDefaultStopSet
public static CharArraySet getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
createComponents
protected Analyzer.TokenStreamComponents createComponents(java.lang.String fieldName)
Returns a (possibly reused)TokenStream
which tokenizes all the text in the providedReader
.- Specified by:
createComponents
in classAnalyzer
- Parameters:
fieldName
- the name of the fields content passed to theAnalyzer.TokenStreamComponents
sink as a reader- Returns:
- A
TokenStream
built from aStandardTokenizer
filtered withLowerCaseFilter
,StopFilter
,SetKeywordMarkerFilter
if a stem exclusion set is provided,StemmerOverrideFilter
, andSnowballFilter
-
normalize
protected TokenStream normalize(java.lang.String fieldName, TokenStream in)
Description copied from class:Analyzer
Wrap the givenTokenStream
in order to apply normalization filters. The default implementation returns theTokenStream
as-is. This is used byAnalyzer.normalize(String, String)
.
-
-