Redis 作为矢量数据库快速入门指南

了解如何将 Redis 用作矢量数据库

本快速入门指南可帮助您：

  1. 了解什么是矢量数据库
  2. 创建 Redis 矢量数据库
  3. 创建向量嵌入并存储向量
  4. 查询数据并执行向量搜索
注意：

本指南使用 RedisVL、 这是 Redis 的 Python 客户端库，高度专门用于 向量处理。您可能还对向量查询示例感兴趣 对于我们的其他客户端库：

了解矢量数据库

数据通常是非结构化的，这意味着它不是由定义明确的架构描述的。非结构化数据的示例包括文本段落、图像、视频或音频。存储和搜索非结构化数据的一种方法是使用向量嵌入。

什么是向量？在机器学习和 AI 中，向量是表示数据的数字序列。它们是模型的输入和输出，以数字形式封装基础信息。向量将非结构化数据（如文本、图像、视频和音频）转换为机器学习模型可以处理的格式。

  • 为什么它们很重要？向量捕获数据中固有的复杂模式和语义含义，使其成为适用于各种应用的强大工具。它们使机器学习模型能够更有效地理解和作非结构化数据。
  • 增强传统搜索。传统的关键字或词法搜索依赖于单词或短语的精确匹配，这可能会有限制。相比之下，向量搜索或语义搜索利用向量嵌入中捕获的丰富信息。通过将数据映射到向量空间，相似的项目会根据它们的含义彼此靠近。这种方法允许更准确和有意义的搜索结果，因为它考虑查询的上下文和语义内容，而不仅仅是使用的确切单词。

创建 Redis 矢量数据库

您可以将 Redis Stack 用作矢量数据库。它允许您：

  • 将向量和关联的元数据存储在哈希或 JSON 文档中
  • 创建和配置用于搜索的二级索引
  • 执行向量搜索
  • 更新向量和元数据
  • 删除和清理

最简单的入门方法是使用 Redis Cloud：

  1. 创建一个免费帐户。

  2. 按照说明创建免费数据库。

这个免费的 Redis Cloud 数据库开箱即用，具有所有 Redis Stack 功能。

您也可以使用安装指南在本地计算机上安装 Redis Stack。

您需要为 Redis 服务器配置以下功能：JSON 和 Search and query。

安装所需的 Python 软件包

创建 Python 虚拟环境并使用pip:

  • redis：您可以找到有关redis-py客户端库。
  • pandas：Pandas 是一个数据分析库。
  • sentence-transformers：您将使用 SentenceTransformers 框架在全文上生成嵌入。
  • tabulate:pandas使用tabulate来渲染 Markdown。

您还需要在 Python 代码中导入以下内容：

连接

连接到 Redis。默认情况下，Redis 返回二进制响应。要解码它们，您需要将decode_responsesparameter 设置为True:


提示：
您可以从 Redis Cloud 数据库配置页面复制并粘贴连接详细信息，而不是使用本地 Redis Stack 服务器。以下是托管在 AWS 区域中的云数据库的示例连接字符串us-east-1并在端口 16379 上侦听：redis-16379.c283.us-east-1-4.ec2.cloud.redislabs.com:16379.连接字符串的格式为host:port.您还必须复制并粘贴 Cloud 数据库的用户名和密码。用于连接默认用户的代码行将更改为client = redis.Redis(host="redis-16379.c283.us-east-1-4.ec2.cloud.redislabs.com", port=16379, password="your_password_here", decode_responses=True).

准备 demo 数据集

本快速入门指南还使用 bikes 数据集。下面是其中的示例文档：

{
  "model": "Jigger",
  "brand": "Velorim",
  "price": 270,
  "type": "Kids bikes",
  "specs": {
    "material": "aluminium",
    "weight": "10"
  },
  "description": "Small and powerful, the Jigger is the best ride for the smallest of tikes! ..."
}

The description field contains free-form text descriptions of bikes and will be used to create vector embeddings.

1. Fetch the demo data

You need to first fetch the demo dataset as a JSON array:

Inspect the structure of one of the bike JSON documents:

2. Store the demo data in Redis

Now iterate over the bikes array to store the data as JSON documents in Redis by using the JSON.SET command. The below code uses a pipeline to minimize the network round-trip times:

Once loaded, you can retrieve a specific attribute from one of the JSON documents in Redis using a JSONPath expression:

3. Select a text embedding model

HuggingFace has a large catalog of text embedding models that are locally servable through the SentenceTransformers framework. Here we use the MS MARCO model that is widely used in search engines, chatbots, and other AI applications.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer('msmarco-distilbert-base-v4')

4. Generate text embeddings

Iterate over all the Redis keys with the prefix bikes::

Use the keys as input to the JSON.MGET command, along with the $.description field, to collect the descriptions in a list. Then, pass the list of descriptions to the .encode() method:

Insert the vectorized descriptions to the bike documents in Redis using the JSON.SET command. The following command inserts a new field into each of the documents under the JSONPath $.description_embeddings. Once again, do this using a pipeline to avoid unnecessary network round-trips:

Inspect one of the updated bike documents using the JSON.GET command:

Note:
When storing a vector embedding within a JSON document, the embedding is stored as a JSON array. In the example above, the array was shortened considerably for the sake of readability.

Create an index

1. Create an index with a vector field

You must create an index to query document metadata or to perform vector searches. Use the FT.CREATE command:

Here is a breakdown of the VECTOR field definition:

  • $.description_embeddings AS vector: The vector field's JSON path and its field alias vector.
  • FLAT: Specifies the indexing method, which is either a flat index or a hierarchical navigable small world graph (HNSW).
  • TYPE FLOAT32: Sets the float precision of a vector component, in this case a 32-bit floating point number.
  • DIM 768: The length or dimension of the embeddings, determined by the chosen embedding model.
  • DISTANCE_METRIC COSINE: The chosen distance function: cosine distance.

You can find further details about all these options in the vector reference documentation.

2. Check the state of the index

As soon as you execute the FT.CREATE command, the indexing process runs in the background. In a short time, all JSON documents should be indexed and ready to be queried. To validate that, you can use the FT.INFO command, which provides details and statistics about the index. Of particular interest are the number of documents successfully indexed and the number of failures:

Perform vector searches

This quick start guide focuses on vector search. However, you can learn more about how to query based on document metadata in the document database quick start guide.

1. Embed your queries

The following code snippet shows a list of text queries you will use to perform vector search in Redis:

First, encode each input query as a vector embedding using the same SentenceTransformers model:


Tip:
It is vital that you use the same embedding model to embed your queries as you did your documents. Using a different model will result in poor semantic search results or error.

The KNN algorithm calculates the distance between the query vector and each vector in Redis based on the chosen distance function. It then returns the top K items with the smallest distances to the query vector. These are the most semantically similar items.

Now construct a query to do just that:

query = (
    Query('(*)=>[KNN 3 @vector $query_vector AS vector_score]')
     .sort_by('vector_score')
     .return_fields('vector_score', 'id', 'brand', 'model', 'description')
     .dialect(2)
)

Let's break down the above query template:

  • The filter expression (*) means all. In other words, no filtering was applied. You could replace it with an expression that filters by additional metadata.
  • The KNN part of the query searches for the top 3 nearest neighbors.
  • The query vector must be passed in as the param query_vector.
  • The distance to the query vector is returned as vector_score.
  • The results are sorted by this vector_score.
  • Finally, it returns the fields vector_score, id, brand, model, and description for each result.
Note:
To utilize a vector query with the FT.SEARCH command, you must specify DIALECT 2 or greater.

You must pass the vectorized query as a byte array with the param name query_vector. The following code creates a Python NumPy array from the query vector and converts it into a compact, byte-level representation that can be passed as a parameter to the query:

client.ft('idx:bikes_vss').search(
    query,
    {
      'query_vector': np.array(encoded_query, dtype=np.float32).tobytes()
    }
).docs

With the template for the query in place, you can execute all queries in a loop. Notice that the script calculates the vector_score for each result as 1 - doc.vector_score. Because the cosine distance is used as the metric, the items with the smallest distance are closer and, therefore, more similar to the query.

Then, loop over the matched documents and create a list of results that can be converted into a Pandas table to visualize the results:

The query results show the individual queries' top three matches (our K parameter) along with the bike's id, brand, and model for each query.

For example, for the query "Best Mountain bikes for kids", the highest similarity score (0.54) and, therefore the closest match was the 'Nord' brand 'Chook air 5' bike model, described as:

The Chook Air 5 gives kids aged six years and older a durable and uberlight mountain bike for their first experience on tracks and easy cruising through forests and fields. The lower top tube makes it easy to mount and dismount in any situation, giving your kids greater safety on the trails. The Chook Air 5 is the perfect intro to mountain biking.

From the description, this bike is an excellent match for younger children, and the embeddings accurately captured the semantics of the description.

query score id brand model description
Best Mountain bikes for kids 0.54 bikes:003 Nord Chook air 5 The Chook Air 5 gives kids aged six years and older a durable and uberlight mountain bike for their first experience on tracks and easy cruising through forests and fields. The lower top tube makes it easy to mount and dismount in any situation, giving your kids greater safety on the trails. The Chook Air 5 is the perfect intro to mountain biking.
0.51 bikes:010 nHill Summit This budget mountain bike from nHill performs well both on bike paths and on the trail. The fork with 100mm of travel absorbs rough terrain. Fat Kenda Booster tires give you grip in corners and on wet trails. The Shimano Tourney drivetrain offered enough gears for finding a comfortable pace to ride uphill, and the Tektro hydraulic disc brakes break smoothly. Whether you want an affordable bike that you can take to work, but also take trail riding on the weekends or you’re just after a stable,...
0.46 bikes:001 Velorim Jigger Small and powerful, the Jigger is the best ride for the smallest of tikes! This is the tiniest kids’ pedal bike on the market available without a coaster brake, the Jigger is the vehicle of choice for the rare tenacious little rider raring to go. We say rare because this smokin’ little bike is not ideal for a nervous first-time rider, but it’s a true giddy up for a true speedster. The Jigger is a 12 inch lightweight kids bicycle and it will meet your little one’s need for speed. It’s a single...

Next steps

  1. You can learn more about the query options, such as filters and vector range queries, by reading the vector reference documentation.
  2. The complete Redis Query Engine documentation might be interesting for you.
  3. If you want to follow the code examples more interactively, then you can use the Jupyter notebook that inspired this quick start guide.
  4. If you want to see more advanced examples of a Redis vector database in action, visit the Redis AI Resources page on GitHub.

Continue learning with Redis University

See the Redis vector database course for learning modules.

RATE THIS PAGE
Back to top ↑