索引和查询向量

了解如何使用 Redis 为向量嵌入编制索引和查询

Redis 查询引擎允许您在哈希或 JSON 对象中为向量字段编制索引（有关更多信息，请参阅向量参考页面）。 除其他外，vector fields 可以存储文本嵌入，这些嵌入是 AI 生成的 vector 文本片段中语义信息的表示形式。两个嵌入向量之间的向量距离表示它们在语义上的相似程度。通过比较 从某些查询文本生成的嵌入与存储在 hash 中的嵌入的相似性 或 JSON 字段，Redis 可以检索在术语方面与查询紧密匹配的文档 的含义。

在下面的示例中，我们使用huggingfaceembedder软件包LinGoose框架生成用于存储和索引的向量嵌入向量 Redis 查询引擎。

初始化

使用以下命令启动新的 Go 模块：

go mod init vecexample

Then, in your module folder, install go-redis and the huggingfaceembedder package:

go get github.com/redis/go-redis/v9
go get github.com/henomis/lingoose/embedder/huggingface

Add the following imports to your module's main program file:

package main

import (
	"context"
	"encoding/binary"
	"fmt"
	"math"

	huggingfaceembedder "github.com/henomis/lingoose/embedder/huggingface"
	"github.com/redis/go-redis/v9"
)

You must also create a HuggingFace account and add a new access token to use the embedding model. See the HuggingFace docs to learn how to create and manage access tokens. Note that the account and the all-MiniLM-L6-v2 model that we will use to produce the embeddings for this example are both available for free.

Add a helper function

The huggingfaceembedder model outputs the embeddings as a []float32 array. If you are storing your documents as hash objects (as we are in this example), then you must convert this array to a byte string before adding it as a hash field. In this example, we will use the function below to produce the byte string:

func floatsToBytes(fs []float32) []byte {
	buf := make([]byte, len(fs)*4)

	for i, f := range fs {
		u := math.Float32bits(f)
		binary.NativeEndian.PutUint32(buf[i*4:], u)
	}

	return buf
}

Note that if you are using JSON objects to store your documents instead of hashes, then you should store the []float32 array directly without first converting it to a byte string.

Create the index

In the main() function, connect to Redis and delete any index previously created with the name vector_idx:

ctx := context.Background()
rdb := redis.NewClient(&redis.Options{
    Addr:     "localhost:6379",
    Password: "", // no password docs
    DB:       0,  // use default DB
    Protocol: 2,
})

rdb.FTDropIndexWithArgs(ctx,
    "vector_idx",
    &redis.FTDropIndexOptions{
        DeleteDocs: true,
    },
)

Next, create the index. The schema in the example below specifies hash objects for storage and includes three fields: the text content to index, a tag field to represent the "genre" of the text, and the embedding vector generated from the original text content. The embedding field specifies HNSW indexing, the L2 vector distance metric, Float32 values to represent the vector's components, and 384 dimensions, as required by the all-MiniLM-L6-v2 embedding model.

_, err := rdb.FTCreate(ctx,
    "vector_idx",
    &redis.FTCreateOptions{
        OnHash: true,
        Prefix: []any{"doc:"},
    },
    &redis.FieldSchema{
        FieldName: "content",
        FieldType: redis.SearchFieldTypeText,
    },
    &redis.FieldSchema{
        FieldName: "genre",
        FieldType: redis.SearchFieldTypeTag,
    },
    &redis.FieldSchema{
        FieldName: "embedding",
        FieldType: redis.SearchFieldTypeVector,
        VectorArgs: &redis.FTVectorArgs{
            HNSWOptions: &redis.FTHNSWOptions{
                Dim:            384,
                DistanceMetric: "L2",
                Type:           "FLOAT32",
            },
        },
    },
).Result()

if err != nil {
    panic(err)
}

Create an embedder instance

You need an instance of the huggingfaceembedder class to generate the embeddings. Use the code below to create an instance that uses the sentence-transformers/all-MiniLM-L6-v2 model, passing your HuggingFace access token to the WithToken() method.

hf := huggingfaceembedder.New().
		WithToken("<your-access-token>").
		WithModel("sentence-transformers/all-MiniLM-L6-v2")

Add data

You can now supply the data objects, which will be indexed automatically when you add them with hset(), as long as you use the doc: prefix specified in the index definition.

Use the Embed() method of huggingfacetransformer as shown below to create the embeddings that represent the content fields. This method takes an array of strings and outputs a corresponding array of Embedding objects. Use the ToFloat32() method of Embedding to produce the array of float values that we need, and use the floatsToBytes() function we defined above to convert this array to a byte string.

sentences := []string{
    "That is a very happy person",
    "That is a happy dog",
    "Today is a sunny day",
}

tags := []string{
    "persons", "pets", "weather",
}

embeddings, err := hf.Embed(ctx, sentences)

if err != nil {
    panic(err)
}

for i, emb := range embeddings {
    buffer := floatsToBytes(emb.ToFloat32())

    if err != nil {
        panic(err)
    }

    _, err = rdb.HSet(ctx,
        fmt.Sprintf("doc:%v", i),
        map[string]any{
            "content":   sentences[i],
            "genre":     tags[i],
            "embedding": buffer,
        },
    ).Result()

    if err != nil {
        panic(err)
    }
}

Run a query

After you have created the index and added the data, you are ready to run a query. To do this, you must create another embedding vector from your chosen query text. Redis calculates the similarity between the query vector and each embedding vector in the index as it runs the query. It then ranks the results in order of this numeric similarity value.

The code below creates the query embedding using Embed(), as with the indexing, and passes it as a parameter when the query executes (see Vector search for more information about using query parameters with embeddings).

queryEmbedding, err := hf.Embed(ctx, []string{
    "That is a happy person",
})

if err != nil {
    panic(err)
}

buffer := floatsToBytes(queryEmbedding[0].ToFloat32())

if err != nil {
    panic(err)
}

results, err := rdb.FTSearchWithArgs(ctx,
    "vector_idx",
    "*=>[KNN 3 @embedding $vec AS vector_distance]",
    &redis.FTSearchOptions{
        Return: []redis.FTSearchReturn{
            {FieldName: "vector_distance"},
            {FieldName: "content"},
        },
        DialectVersion: 2,
        Params: map[string]any{
            "vec": buffer,
        },
    },
).Result()

if err != nil {
    panic(err)
}

for _, doc := range results.Docs {
    fmt.Printf(
        "ID: %v, Distance:%v, Content:'%v'\n",
        doc.ID, doc.Fields["vector_distance"], doc.Fields["content"],
    )
}

The code is now ready to run, but note that it may take a while to complete when you run it for the first time (which happens because huggingfacetransformer must download the all-MiniLM-L6-v2 model data before it can generate the embeddings). When you run the code, it outputs the following text:

ID: doc:0, Distance:0.114169843495, Content:'That is a very happy person'
ID: doc:1, Distance:0.610845327377, Content:'That is a happy dog'
ID: doc:2, Distance:1.48624765873, Content:'Today is a sunny day'

The results are ordered according to the value of the vector_distance field, with the lowest distance indicating the greatest similarity to the query. As you would expect, the result for doc:0 with the content text "That is a very happy person" is the result that is most similar in meaning to the query text "That is a happy person".

Learn more

See Vector search for more information about the indexing options, distance metrics, and query format for vectors.

RATE THIS PAGE
Back to top ↑