Class Lucene90CompressingStoredFieldsWriter
- java.lang.Object
-
- org.apache.lucene.codecs.StoredFieldsWriter
-
- org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Accountable
public final class Lucene90CompressingStoredFieldsWriter extends StoredFieldsWriter
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
Lucene90CompressingStoredFieldsWriter.CompressingStoredFieldsMergeSub
private static class
Lucene90CompressingStoredFieldsWriter.MergeStrategy
-
Nested classes/interfaces inherited from class org.apache.lucene.codecs.StoredFieldsWriter
StoredFieldsWriter.MergeVisitor
-
-
Field Summary
Fields Modifier and Type Field Description private ByteBuffersDataOutput
bufferedDocs
(package private) static boolean
BULK_MERGE_ENABLED
(package private) static java.lang.String
BULK_MERGE_ENABLED_SYSPROP
(package private) static int
BYTE_ARR
private int
chunkSize
private CompressionMode
compressionMode
private Compressor
compressor
(package private) static long
DAY
(package private) static int
DAY_ENCODING
private int
docBase
private int[]
endOffsets
static java.lang.String
FIELDS_EXTENSION
Extension of stored fields fileprivate IndexOutput
fieldsStream
(package private) static long
HOUR
(package private) static int
HOUR_ENCODING
static java.lang.String
INDEX_CODEC_NAME
Codec name for the index.static java.lang.String
INDEX_EXTENSION
Extension of stored fields indexprivate FieldsIndexWriter
indexWriter
private int
maxDocsPerChunk
static java.lang.String
META_EXTENSION
Extension of stored fields meta(package private) static int
META_VERSION_START
private IndexOutput
metaStream
(package private) static long
NEGATIVE_ZERO_DOUBLE
(package private) static int
NEGATIVE_ZERO_FLOAT
private int
numBufferedDocs
private long
numChunks
private long
numDirtyChunks
private long
numDirtyDocs
(package private) static int
NUMERIC_DOUBLE
(package private) static int
NUMERIC_FLOAT
(package private) static int
NUMERIC_INT
(package private) static int
NUMERIC_LONG
private int[]
numStoredFields
private int
numStoredFieldsInDoc
(package private) static long
SECOND
(package private) static int
SECOND_ENCODING
private java.lang.String
segment
(package private) static int
STRING
(package private) static int
TYPE_BITS
(package private) static int
TYPE_MASK
(package private) static int
VERSION_CURRENT
(package private) static int
VERSION_START
-
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
-
Constructor Summary
Constructors Constructor Description Lucene90CompressingStoredFieldsWriter(Directory directory, SegmentInfo si, java.lang.String segmentSuffix, IOContext context, java.lang.String formatName, CompressionMode compressionMode, int chunkSize, int maxDocsPerChunk, int blockShift)
Sole constructor.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
private void
copyChunks(MergeState mergeState, Lucene90CompressingStoredFieldsWriter.CompressingStoredFieldsMergeSub sub, int fromDocID, int toDocID)
private void
copyOneDoc(Lucene90CompressingStoredFieldsReader reader, int docID)
void
finish(int numDocs)
Called beforeStoredFieldsWriter.close()
, passing in the number of documents that were written.void
finishDocument()
Called when a document and all its fields have been added.private void
flush(boolean force)
private Lucene90CompressingStoredFieldsWriter.MergeStrategy
getMergeStrategy(MergeState mergeState, MatchingReaders matchingReaders, int readerIndex)
int
merge(MergeState mergeState)
Merges in the stored fields from the readers inmergeState
.long
ramBytesUsed()
Return the memory usage of this object in bytes.private static void
saveInts(int[] values, int length, DataOutput out)
void
startDocument()
Called before writing the stored fields of the document.(package private) boolean
tooDirty(Lucene90CompressingStoredFieldsReader candidate)
Returns true if we should recompress this reader, even though we could bulk merge compressed dataprivate boolean
triggerFlush()
void
writeField(FieldInfo info, IndexableField field)
Writes a single stored field.private void
writeHeader(int docBase, int numBufferedDocs, int[] numStoredFields, int[] lengths, boolean sliced, boolean dirtyChunk)
(package private) static void
writeTLong(DataOutput out, long l)
Writes a long in a variable-length format.(package private) static void
writeZDouble(DataOutput out, double d)
Writes a float in a variable-length format.(package private) static void
writeZFloat(DataOutput out, float f)
Writes a float in a variable-length format.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
FIELDS_EXTENSION
public static final java.lang.String FIELDS_EXTENSION
Extension of stored fields file- See Also:
- Constant Field Values
-
INDEX_EXTENSION
public static final java.lang.String INDEX_EXTENSION
Extension of stored fields index- See Also:
- Constant Field Values
-
META_EXTENSION
public static final java.lang.String META_EXTENSION
Extension of stored fields meta- See Also:
- Constant Field Values
-
INDEX_CODEC_NAME
public static final java.lang.String INDEX_CODEC_NAME
Codec name for the index.- See Also:
- Constant Field Values
-
STRING
static final int STRING
- See Also:
- Constant Field Values
-
BYTE_ARR
static final int BYTE_ARR
- See Also:
- Constant Field Values
-
NUMERIC_INT
static final int NUMERIC_INT
- See Also:
- Constant Field Values
-
NUMERIC_FLOAT
static final int NUMERIC_FLOAT
- See Also:
- Constant Field Values
-
NUMERIC_LONG
static final int NUMERIC_LONG
- See Also:
- Constant Field Values
-
NUMERIC_DOUBLE
static final int NUMERIC_DOUBLE
- See Also:
- Constant Field Values
-
TYPE_BITS
static final int TYPE_BITS
-
TYPE_MASK
static final int TYPE_MASK
-
VERSION_START
static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_CURRENT
static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
META_VERSION_START
static final int META_VERSION_START
- See Also:
- Constant Field Values
-
segment
private final java.lang.String segment
-
indexWriter
private FieldsIndexWriter indexWriter
-
metaStream
private IndexOutput metaStream
-
fieldsStream
private IndexOutput fieldsStream
-
compressor
private Compressor compressor
-
compressionMode
private final CompressionMode compressionMode
-
chunkSize
private final int chunkSize
-
maxDocsPerChunk
private final int maxDocsPerChunk
-
bufferedDocs
private final ByteBuffersDataOutput bufferedDocs
-
numStoredFields
private int[] numStoredFields
-
endOffsets
private int[] endOffsets
-
docBase
private int docBase
-
numBufferedDocs
private int numBufferedDocs
-
numChunks
private long numChunks
-
numDirtyChunks
private long numDirtyChunks
-
numDirtyDocs
private long numDirtyDocs
-
numStoredFieldsInDoc
private int numStoredFieldsInDoc
-
NEGATIVE_ZERO_FLOAT
static final int NEGATIVE_ZERO_FLOAT
-
NEGATIVE_ZERO_DOUBLE
static final long NEGATIVE_ZERO_DOUBLE
-
SECOND
static final long SECOND
- See Also:
- Constant Field Values
-
HOUR
static final long HOUR
- See Also:
- Constant Field Values
-
DAY
static final long DAY
- See Also:
- Constant Field Values
-
SECOND_ENCODING
static final int SECOND_ENCODING
- See Also:
- Constant Field Values
-
HOUR_ENCODING
static final int HOUR_ENCODING
- See Also:
- Constant Field Values
-
DAY_ENCODING
static final int DAY_ENCODING
- See Also:
- Constant Field Values
-
BULK_MERGE_ENABLED_SYSPROP
static final java.lang.String BULK_MERGE_ENABLED_SYSPROP
-
BULK_MERGE_ENABLED
static final boolean BULK_MERGE_ENABLED
-
-
Constructor Detail
-
Lucene90CompressingStoredFieldsWriter
Lucene90CompressingStoredFieldsWriter(Directory directory, SegmentInfo si, java.lang.String segmentSuffix, IOContext context, java.lang.String formatName, CompressionMode compressionMode, int chunkSize, int maxDocsPerChunk, int blockShift) throws java.io.IOException
Sole constructor.- Throws:
java.io.IOException
-
-
Method Detail
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Specified by:
close
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
startDocument
public void startDocument() throws java.io.IOException
Description copied from class:StoredFieldsWriter
Called before writing the stored fields of the document.StoredFieldsWriter.writeField(FieldInfo, IndexableField)
will be called for each stored field. Note that this is called even if the document has no stored fields.- Specified by:
startDocument
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
finishDocument
public void finishDocument() throws java.io.IOException
Description copied from class:StoredFieldsWriter
Called when a document and all its fields have been added.- Overrides:
finishDocument
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
saveInts
private static void saveInts(int[] values, int length, DataOutput out) throws java.io.IOException
- Throws:
java.io.IOException
-
writeHeader
private void writeHeader(int docBase, int numBufferedDocs, int[] numStoredFields, int[] lengths, boolean sliced, boolean dirtyChunk) throws java.io.IOException
- Throws:
java.io.IOException
-
triggerFlush
private boolean triggerFlush()
-
flush
private void flush(boolean force) throws java.io.IOException
- Throws:
java.io.IOException
-
writeField
public void writeField(FieldInfo info, IndexableField field) throws java.io.IOException
Description copied from class:StoredFieldsWriter
Writes a single stored field.- Specified by:
writeField
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
writeZFloat
static void writeZFloat(DataOutput out, float f) throws java.io.IOException
Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.ZFloat --> Header, Bytes*?
- Header -->
Uint8
. When it is equal to 0xFF then the value is negative and stored in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 3 bytes need to be read to complete it. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
java.io.IOException
- Header -->
-
writeZDouble
static void writeZDouble(DataOutput out, double d) throws java.io.IOException
Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.ZFloat --> Header, Bytes*?
- Header -->
Uint8
. When it is equal to 0xFF then the value is negative and stored in the next 8 bytes. When it is equal to 0xFE then the value is stored as a float in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 7 bytes need to be read to complete it. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
java.io.IOException
- Header -->
-
writeTLong
static void writeTLong(DataOutput out, long l) throws java.io.IOException
Writes a long in a variable-length format. Writes between one and ten bytes. Small values or values representing timestamps with day, hour or second precision typically require fewer bytes.ZLong --> Header, Bytes*?
- Header --> The first two bits indicate the compression scheme:
- 00 - uncompressed
- 01 - multiple of 1000 (second)
- 10 - multiple of 3600000 (hour)
- 11 - multiple of 86400000 (day)
zigzag-decode
it and finally multiply by the multiple corresponding to the compression scheme. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
java.io.IOException
- Header --> The first two bits indicate the compression scheme:
-
finish
public void finish(int numDocs) throws java.io.IOException
Description copied from class:StoredFieldsWriter
Called beforeStoredFieldsWriter.close()
, passing in the number of documents that were written. Note that this is intentionally redundant (equivalent to the number of calls toStoredFieldsWriter.startDocument()
, but a Codec should check that this is the case to detect the JRE bug described in LUCENE-1282.- Specified by:
finish
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
copyOneDoc
private void copyOneDoc(Lucene90CompressingStoredFieldsReader reader, int docID) throws java.io.IOException
- Throws:
java.io.IOException
-
copyChunks
private void copyChunks(MergeState mergeState, Lucene90CompressingStoredFieldsWriter.CompressingStoredFieldsMergeSub sub, int fromDocID, int toDocID) throws java.io.IOException
- Throws:
java.io.IOException
-
merge
public int merge(MergeState mergeState) throws java.io.IOException
Description copied from class:StoredFieldsWriter
Merges in the stored fields from the readers inmergeState
. The default implementation skips over deleted documents, and usesStoredFieldsWriter.startDocument()
,StoredFieldsWriter.writeField(FieldInfo, IndexableField)
, andStoredFieldsWriter.finish(int)
, returning the number of documents that were written. Implementations can override this method for more sophisticated merging (bulk-byte copying, etc).- Overrides:
merge
in classStoredFieldsWriter
- Throws:
java.io.IOException
-
tooDirty
boolean tooDirty(Lucene90CompressingStoredFieldsReader candidate)
Returns true if we should recompress this reader, even though we could bulk merge compressed dataThe last chunk written for a segment is typically incomplete, so without recompressing, in some worst-case situations (e.g. frequent reopen with tiny flushes), over time the compression ratio can degrade. This is a safety switch.
-
getMergeStrategy
private Lucene90CompressingStoredFieldsWriter.MergeStrategy getMergeStrategy(MergeState mergeState, MatchingReaders matchingReaders, int readerIndex)
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.
-
-