Package org.apache.lucene.index
Class DocumentsWriterDeleteQueue
- java.lang.Object
-
- org.apache.lucene.index.DocumentsWriterDeleteQueue
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Accountable
final class DocumentsWriterDeleteQueue extends java.lang.Object implements Accountable, java.io.Closeable
DocumentsWriterDeleteQueue
is a non-blocking linked pending deletes queue. In contrast to other queue implementation we only maintain the tail of the queue. A delete queue is always used in a context of a set of DWPTs and a global delete pool. Each of the DWPT and the global pool need to maintain their 'own' head of the queue (as a DeleteSlice instance perDocumentsWriterPerThread
). The difference between the DWPT and the global pool is that the DWPT starts maintaining a head once it has added its first document since for its segments private deletes only the deletes after that document are relevant. The global pool instead starts maintaining the head once this instance is created by taking the sentinel instance as its initial head.Since each
DocumentsWriterDeleteQueue.DeleteSlice
maintains its own head and the list is only single linked the garbage collector takes care of pruning the list for us. All nodes in the list that are still relevant should be either directly or indirectly referenced by one of the DWPT's privateDocumentsWriterDeleteQueue.DeleteSlice
or by the globalBufferedUpdates
slice.Each DWPT as well as the global delete pool maintain their private DeleteSlice instance. In the DWPT case updating a slice is equivalent to atomically finishing the document. The slice update guarantees a "happens before" relationship to all other updates in the same indexing session. When a DWPT updates a document it:
- consumes a document and finishes its processing
- updates its private
DocumentsWriterDeleteQueue.DeleteSlice
either by callingupdateSlice(DeleteSlice)
oradd(Node, DeleteSlice)
(if the document has a delTerm) - applies all deletes in the slice to its private
BufferedUpdates
and resets it - increments its internal document id
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static class
DocumentsWriterDeleteQueue.DeleteSlice
private static class
DocumentsWriterDeleteQueue.DocValuesUpdatesNode
(package private) static class
DocumentsWriterDeleteQueue.Node<T>
private static class
DocumentsWriterDeleteQueue.QueryArrayNode
private static class
DocumentsWriterDeleteQueue.TermArrayNode
private static class
DocumentsWriterDeleteQueue.TermNode
-
Field Summary
Fields Modifier and Type Field Description private boolean
advanced
private boolean
closed
(package private) long
generation
private BufferedUpdates
globalBufferedUpdates
(package private) java.util.concurrent.locks.ReentrantLock
globalBufferLock
private DocumentsWriterDeleteQueue.DeleteSlice
globalSlice
Used to record deletes against all prior (already written to disk) segments.private InfoStream
infoStream
private long
maxSeqNo
private java.util.concurrent.atomic.AtomicLong
nextSeqNo
Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations.private java.util.function.LongSupplier
previousMaxSeqId
private long
startSeqNo
private DocumentsWriterDeleteQueue.Node<?>
tail
-
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
-
Constructor Summary
Constructors Modifier Constructor Description (package private)
DocumentsWriterDeleteQueue(InfoStream infoStream)
private
DocumentsWriterDeleteQueue(InfoStream infoStream, long generation, long startSeqNo, java.util.function.LongSupplier previousMaxSeqId)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) long
add(DocumentsWriterDeleteQueue.Node<?> newNode)
(package private) long
add(DocumentsWriterDeleteQueue.Node<?> deleteNode, DocumentsWriterDeleteQueue.DeleteSlice slice)
invariant for document update(package private) long
addDelete(Term... terms)
(package private) long
addDelete(Query... queries)
(package private) long
addDocValuesUpdates(DocValuesUpdate... updates)
(package private) DocumentsWriterDeleteQueue
advanceQueue(int maxNumPendingOps)
Advances the queue to the next queue on flush.(package private) boolean
anyChanges()
(package private) void
clear()
void
close()
private void
ensureOpen()
(package private) FrozenBufferedUpdates
freezeGlobalBuffer(DocumentsWriterDeleteQueue.DeleteSlice callerSlice)
private FrozenBufferedUpdates
freezeGlobalBufferInternal(DocumentsWriterDeleteQueue.Node<?> currentTail)
int
getBufferedUpdatesTermsSize()
(package private) long
getLastSequenceNumber()
(package private) long
getMaxCompletedSeqNo()
Returns the maximum completed seq no for this queue.(package private) long
getMaxSeqNo()
Returns the maximum sequence number for this queue.long
getNextSequenceNumber()
private static java.util.function.LongSupplier
getPrevMaxSeqIdSupplier(java.util.concurrent.atomic.AtomicLong nextSeqNo)
(package private) boolean
isAdvanced()
Returnstrue
if it was advanced.boolean
isOpen()
(package private) FrozenBufferedUpdates
maybeFreezeGlobalBuffer()
This may freeze the global buffer unless the delete queue has already been closed.(package private) static DocumentsWriterDeleteQueue.Node<DocValuesUpdate[]>
newNode(DocValuesUpdate... updates)
(package private) static DocumentsWriterDeleteQueue.Node<Term>
newNode(Term term)
(package private) DocumentsWriterDeleteQueue.DeleteSlice
newSlice()
int
numGlobalTermDeletes()
long
ramBytesUsed()
Return the memory usage of this object in bytes.(package private) void
skipSequenceNumbers(long jump)
Inserts a gap in the sequence numbers.java.lang.String
toString()
(package private) void
tryApplyGlobalSlice()
(package private) long
updateSlice(DocumentsWriterDeleteQueue.DeleteSlice slice)
Negative result means there were new deletes since we last applied(package private) boolean
updateSliceNoSeqNo(DocumentsWriterDeleteQueue.DeleteSlice slice)
Just like updateSlice, but does not assign a sequence number-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
tail
private volatile DocumentsWriterDeleteQueue.Node<?> tail
-
closed
private volatile boolean closed
-
globalSlice
private final DocumentsWriterDeleteQueue.DeleteSlice globalSlice
Used to record deletes against all prior (already written to disk) segments. Whenever any segment flushes, we bundle up this set of deletes and insert into the buffered updates stream before the newly flushed segment(s).
-
globalBufferedUpdates
private final BufferedUpdates globalBufferedUpdates
-
globalBufferLock
final java.util.concurrent.locks.ReentrantLock globalBufferLock
-
generation
final long generation
-
nextSeqNo
private final java.util.concurrent.atomic.AtomicLong nextSeqNo
Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations.
-
infoStream
private final InfoStream infoStream
-
maxSeqNo
private volatile long maxSeqNo
-
startSeqNo
private final long startSeqNo
-
previousMaxSeqId
private final java.util.function.LongSupplier previousMaxSeqId
-
advanced
private boolean advanced
-
-
Constructor Detail
-
DocumentsWriterDeleteQueue
DocumentsWriterDeleteQueue(InfoStream infoStream)
-
DocumentsWriterDeleteQueue
private DocumentsWriterDeleteQueue(InfoStream infoStream, long generation, long startSeqNo, java.util.function.LongSupplier previousMaxSeqId)
-
-
Method Detail
-
addDelete
long addDelete(Query... queries)
-
addDelete
long addDelete(Term... terms)
-
addDocValuesUpdates
long addDocValuesUpdates(DocValuesUpdate... updates)
-
newNode
static DocumentsWriterDeleteQueue.Node<Term> newNode(Term term)
-
newNode
static DocumentsWriterDeleteQueue.Node<DocValuesUpdate[]> newNode(DocValuesUpdate... updates)
-
add
long add(DocumentsWriterDeleteQueue.Node<?> deleteNode, DocumentsWriterDeleteQueue.DeleteSlice slice)
invariant for document update
-
add
long add(DocumentsWriterDeleteQueue.Node<?> newNode)
-
anyChanges
boolean anyChanges()
-
tryApplyGlobalSlice
void tryApplyGlobalSlice()
-
freezeGlobalBuffer
FrozenBufferedUpdates freezeGlobalBuffer(DocumentsWriterDeleteQueue.DeleteSlice callerSlice)
-
maybeFreezeGlobalBuffer
FrozenBufferedUpdates maybeFreezeGlobalBuffer()
This may freeze the global buffer unless the delete queue has already been closed. If the queue has been closed this method will returnnull
-
freezeGlobalBufferInternal
private FrozenBufferedUpdates freezeGlobalBufferInternal(DocumentsWriterDeleteQueue.Node<?> currentTail)
-
newSlice
DocumentsWriterDeleteQueue.DeleteSlice newSlice()
-
updateSlice
long updateSlice(DocumentsWriterDeleteQueue.DeleteSlice slice)
Negative result means there were new deletes since we last applied
-
updateSliceNoSeqNo
boolean updateSliceNoSeqNo(DocumentsWriterDeleteQueue.DeleteSlice slice)
Just like updateSlice, but does not assign a sequence number
-
ensureOpen
private void ensureOpen()
-
isOpen
public boolean isOpen()
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
-
numGlobalTermDeletes
public int numGlobalTermDeletes()
-
clear
void clear()
-
getBufferedUpdatesTermsSize
public int getBufferedUpdatesTermsSize()
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
getNextSequenceNumber
public long getNextSequenceNumber()
-
getLastSequenceNumber
long getLastSequenceNumber()
-
skipSequenceNumbers
void skipSequenceNumbers(long jump)
Inserts a gap in the sequence numbers. This is used by IW during flush or commit to ensure any in-flight threads get sequence numbers inside the gap
-
getMaxCompletedSeqNo
long getMaxCompletedSeqNo()
Returns the maximum completed seq no for this queue.
-
getPrevMaxSeqIdSupplier
private static java.util.function.LongSupplier getPrevMaxSeqIdSupplier(java.util.concurrent.atomic.AtomicLong nextSeqNo)
-
advanceQueue
DocumentsWriterDeleteQueue advanceQueue(int maxNumPendingOps)
Advances the queue to the next queue on flush. This carries over the the generation to the next queue and set thegetMaxSeqNo()
based on the given maxNumPendingOps. This method can only be called once, subsequently the returned queue should be used.- Parameters:
maxNumPendingOps
- the max number of possible concurrent operations that will execute on this queue after it was advanced. This corresponds the the number of DWPTs that own the current queue at the moment when this queue is advanced since each these DWPTs can increment the seqId after we advanced it.- Returns:
- a new queue as a successor of this queue.
-
getMaxSeqNo
long getMaxSeqNo()
Returns the maximum sequence number for this queue. This value will change once this queue is advanced.
-
isAdvanced
boolean isAdvanced()
Returnstrue
if it was advanced.
-
-