Class Lucene91HnswVectorsFormat
- java.lang.Object
-
- org.apache.lucene.codecs.KnnVectorsFormat
-
- org.apache.lucene.backward_codecs.lucene91.Lucene91HnswVectorsFormat
-
- All Implemented Interfaces:
NamedSPILoader.NamedSPI
public class Lucene91HnswVectorsFormat extends KnnVectorsFormat
Lucene 9.1 vector format, which encodes numeric vector values and an optional associated graph connecting the documents having values. The graph is used to power HNSW search. The format consists of three files:.vec (vector data) file
This file stores all the floating-point vector data ordered by field, document ordinal, and vector dimension. The floats are stored in little-endian byte order.
.vex (vector index)
Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:
- For each level:
- For each node:
- [int32] the number of neighbor nodes
- array[int32] the neighbor ordinals
- array[int32] padding from empty integers if the number of neighbors less than the maximum number of connections (maxConn). Padding is equal to ((maxConn-the number of neighbours) * 4) bytes.
- For each node:
.vem (vector metadata) file
For each field:
- [int32] field number
- [int32] vector similarity function ordinal
- [vlong] offset to this field's vectors in the .vec file
- [vlong] length of this field's vectors, in bytes
- [vlong] offset to this field's index in the .vex file
- [vlong] length of this field's index data, in bytes
- [int] dimension of this field's vectors
- [int] the number of documents having values for this field
- [int8] if equals to -1, dense – all documents have values for a field. If equals to 0, sparse – some documents missing values.
- array[int] for sparse case, the docids of documents having vectors, in order
- [int] the maximum number of connections (neigbours) that each node can have
- [int] number of levels in the graph
- Graph nodes by level. For each level
- [int] the number of nodes on this level
- array[int] for levels greater than 0 list of nodes on this level, stored as the the level 0th nodes ordinals.
-
-
Field Summary
Fields Modifier and Type Field Description (package private) int
beamWidth
The number of candidate neighbors to track while searching the graph for each newly inserted node.static int
DEFAULT_BEAM_WIDTH
Default number of the size of the queue maintained while searching during a graph construction.static int
DEFAULT_MAX_CONN
Default number of maximum connections per node(package private) int
maxConn
Controls how many of the nearest neighbor candidates are connected to the new node.(package private) static java.lang.String
META_CODEC_NAME
(package private) static java.lang.String
META_EXTENSION
(package private) static java.lang.String
VECTOR_DATA_CODEC_NAME
(package private) static java.lang.String
VECTOR_DATA_EXTENSION
(package private) static java.lang.String
VECTOR_INDEX_CODEC_NAME
(package private) static java.lang.String
VECTOR_INDEX_EXTENSION
(package private) static int
VERSION_CURRENT
(package private) static int
VERSION_START
-
Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat
EMPTY
-
-
Constructor Summary
Constructors Constructor Description Lucene91HnswVectorsFormat()
A constructor for vectors format with default parametersLucene91HnswVectorsFormat(int maxConn, int beamWidth)
A constructor for vectors format
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description KnnVectorsReader
fieldsReader(SegmentReadState state)
Returns aKnnVectorsReader
to read the vectors from the index.KnnVectorsWriter
fieldsWriter(SegmentWriteState state)
Returns aKnnVectorsWriter
to write the vectors to the index.java.lang.String
toString()
-
Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat
forName, getName
-
-
-
-
Field Detail
-
META_CODEC_NAME
static final java.lang.String META_CODEC_NAME
- See Also:
- Constant Field Values
-
VECTOR_DATA_CODEC_NAME
static final java.lang.String VECTOR_DATA_CODEC_NAME
- See Also:
- Constant Field Values
-
VECTOR_INDEX_CODEC_NAME
static final java.lang.String VECTOR_INDEX_CODEC_NAME
- See Also:
- Constant Field Values
-
META_EXTENSION
static final java.lang.String META_EXTENSION
- See Also:
- Constant Field Values
-
VECTOR_DATA_EXTENSION
static final java.lang.String VECTOR_DATA_EXTENSION
- See Also:
- Constant Field Values
-
VECTOR_INDEX_EXTENSION
static final java.lang.String VECTOR_INDEX_EXTENSION
- See Also:
- Constant Field Values
-
VERSION_START
static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_CURRENT
static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
DEFAULT_MAX_CONN
public static final int DEFAULT_MAX_CONN
Default number of maximum connections per node- See Also:
- Constant Field Values
-
DEFAULT_BEAM_WIDTH
public static final int DEFAULT_BEAM_WIDTH
Default number of the size of the queue maintained while searching during a graph construction.- See Also:
- Constant Field Values
-
maxConn
final int maxConn
Controls how many of the nearest neighbor candidates are connected to the new node. Defaults toDEFAULT_MAX_CONN
. SeeHnswGraph
for more details.
-
beamWidth
final int beamWidth
The number of candidate neighbors to track while searching the graph for each newly inserted node. Defaults to toDEFAULT_BEAM_WIDTH
. SeeHnswGraph
for details.
-
-
Method Detail
-
fieldsWriter
public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws java.io.IOException
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsWriter
to write the vectors to the index.- Specified by:
fieldsWriter
in classKnnVectorsFormat
- Throws:
java.io.IOException
-
fieldsReader
public KnnVectorsReader fieldsReader(SegmentReadState state) throws java.io.IOException
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsReader
to read the vectors from the index.- Specified by:
fieldsReader
in classKnnVectorsFormat
- Throws:
java.io.IOException
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
-