【RAG】【vector_stores042】Jaguar向量存储示例分析

1. 案例目标本案例展示了如何使用Jaguar向量数据库与LlamaIndex集成实现高效的向量存储和检索功能。Jaguar是一个分布式向量数据库具有零移动(ZeroMove)功能可实现即时水平扩展支持多种数据类型和距离度量并提供高级功能如时间衰减搜索和异常检测。2. 技术栈与核心依赖向量数据库Jaguar - 分布式向量数据库向量存储集成llama-index-vector-stores-jaguarHTTP客户端jaguardb-http-client核心框架llama-index数据处理SimpleDirectoryReader索引构建VectorStoreIndex存储上下文StorageContext3. 环境配置3.1 Jaguar服务器设置有两种方式设置Jaguar服务器方法一Dockerdocker pull jaguardb/jaguardb docker run -d -p 8888:8888 -p 8080:8080 --name jaguardb jaguardb/jaguardb pip install -U llama-index pip install -U jaguardb-http-client方法二快速设置(Linux)curl -fsSL http://jaguardb.com/install.sh | sh pip install -U llama-index pip install -U jaguardb-http-client3.2 安装Python依赖%pip install llama-index-vector-stores-jaguar!pip install -U jaguardb-http-client3.3 导入必要的依赖包from llama_index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama_index.core import StorageContextfrom llama_index.vector_stores.jaguar import JaguarVectorStorefrom jaguardb_http_client.JaguarHttpClient import JaguarHttpClient4. Jaguar向量存储特性分布式架构可以存储大量向量支持即时水平扩展多数据类型支持支持嵌入向量、文本、图像、视频、PDF、音频、时间序列和空间数据全主架构允许并行读写操作异常检测可以识别数据集中的异常值RAG支持可以结合LLM和专有实时数据元数据共享跨多个向量索引共享元数据提高数据一致性多种距离度量支持欧几里得、余弦、内积、曼哈顿、切比雪夫、汉明、杰卡德和闵可夫斯基距离时间衰减搜索支持带时间截止和时间衰减效应的相似性搜索5. 案例实现5.1 创建Jaguar向量存储客户端url http://127.0.0.1:8080/fwww/pod vdbstore llamaindex_jaguar_storevector_index vvector_type cosine_fraction_float # vector_type cosine_fraction_short 相比float节省一半内存# vector_type cosine_fraction_byte 相比float节省四分之三内存vector_dimension 1536 # 根据OpenAIEmbedding模型jaguarstore JaguarVectorStore(pod,store,vector_index,vector_type,vector_dimension,url,)参数说明urlHTTP网关服务器的端点podPod(或数据库)名称store向量存储的名称vector_index存储中向量索引的名称vector_type向量索引的属性包括距离度量和存储格式vector_dimension向量的维度5.2 身份验证true_or_false jaguarstore.login()print(flogin result is {true_or_false})客户端必须登录或连接到后端jaguar服务器进行系统安全和用户认证。环境变量JAGUAR_API_KEY或文件$HOME/.jagrc文件必须包含系统管理员颁发的jaguar API密钥。5.3 创建向量存储metadata_str author char(32), category char(16) text_size 1024 jaguarstore.create(metadata_str, text_size)创建一个带有v:text字段(大小为1024字节)的向量存储用于保存文本以及两个额外的元数据字段author和category。5.4 加载文档documents SimpleDirectoryReader(../data/paul_graham/).load_data()print(floading {len(documents)} doument(s))5.5 构建索引### make a storage context using our vector storestorage_context StorageContext.from_defaults(vector_storejaguarstore)### clear all vectors in the vector storejaguarstore.clear()### make an index with the documents,storage contextindex VectorStoreIndex.from_documents(documents, storage_contextstorage_context)### You could add more documents to the vector store:# jaguarstore.add_documents(some_docs)# jaguarstore.add_documents(more_docs, text_tagtag to these documents)### print number of documents in jaguar vector storenum jaguarstore.count()print(fThere are {num} vectors in jaguar vector store)5.6 查询测试query_engine index.as_query_engine()q What did the author do growing up?print(fQuestion: {q})response query_engine.query(q)print(fAnswer: {str(response)}\n)q What did the author do after his time at Viaweb?print(fQuestion: {q})response query_engine.query(q)print(fAnswer: {str(response)})5.7 高级查询选项qkwargs {args: day_cutoff365,day_decay_rate0.01,where: categorystartup or category,}query_engine_filter index.as_query_engine(vector_store_kwargsqkwargs)q What was the authors life style?print(fQuestion: {q})response query_engine_filter.query(q)print(fAnswer: {str(response)})可以通过vector_store_kwargs参数传递额外参数到查询引擎以从jaguar向量存储中选择数据子集day_cutoff超过指定天数的文本将被忽略day_decay_rate相似度分数的每日衰减率where用于过滤元数据的SQL WHERE条件5.8 清理和注销### remove all the data in the vector store if you wantjaguarstore.clear()### delete the whole vector in the database if you wantjaguarstore.drop()### disconnect from jaguar server and cleanup resourcesjaguarstore.logout()6. 案例效果运行查询后系统会返回与问题相关的答案例如Question: What did the author do growing up?Answer: The author mentioned that growing up, they worked on two main things outside of school: writing and programming. They wrote short stories and tried writing programs on an IBM 1401 computer.Question: What did the author do after his time at Viaweb?Answer: After his time at Viaweb, the author started a company to put art galleries online. However, this idea did not turn out to be successful as art galleries did not want to be online.使用高级查询选项时系统会根据设置的过滤条件和时间衰减参数返回相关答案Question: What was the authors life style?Answer: The authors lifestyle involved attending the Accademia as a student and painting still lives in their bedroom at night. They also wrote essays and had a messy life, which they thought would be interesting and encouraging to others.7. 案例实现思路本案例的实现思路如下环境准备安装Jaguar服务器和必要的Python依赖包客户端初始化创建JaguarVectorStore对象配置连接参数和向量属性身份验证使用API密钥登录Jaguar服务器向量存储创建定义元数据字段和文本大小创建向量存储数据准备加载Paul Graham的文章作为示例数据索引构建创建StorageContext将向量存储集成到其中然后基于文档构建VectorStoreIndex查询执行将索引转换为查询引擎执行查询并获取响应高级查询使用vector_store_kwargs参数实现时间衰减和元数据过滤资源清理清理向量存储数据并注销连接关键技术点使用vector_type参数指定向量存储格式可以选择float、short或byte以优化内存使用通过metadata_str定义自定义元数据字段支持更丰富的数据组织使用vector_store_kwargs参数实现高级查询功能如时间衰减和元数据过滤通过login()和logout()方法管理安全连接和资源释放8. 扩展建议多向量类型根据应用场景选择不同的向量存储格式(float/short/byte)以优化内存和性能元数据设计设计更丰富的元数据结构支持复杂的查询和过滤需求时间序列分析利用Jaguar的时间衰减功能构建时间感知的检索系统异常检测集成Jaguar的异常检测能力识别数据中的异常模式多模态数据利用Jaguar对图像、视频等多模态数据的支持构建多模态检索系统分布式扩展利用ZeroMove功能实现系统的水平扩展处理更大规模的数据混合查询结合向量搜索和传统SQL查询实现更复杂的检索需求实时更新实现向量数据的实时更新机制保持数据同步9. 总结Jaguar向量数据库是一个功能强大的分布式向量存储解决方案特别适合需要处理大规模、多类型数据的应用场景。其零移动(ZeroMove)功能使得系统可以轻松实现水平扩展而丰富的时间衰减和异常检测功能则为高级应用提供了基础。通过本案例开发者可以学习到如何配置和初始化Jaguar向量存储如何定义自定义元数据字段和向量存储格式如何实现高级查询功能如时间衰减和元数据过滤如何管理安全连接和资源释放如何利用Jaguar的独特功能构建高级应用这种集成方式为构建大规模、高性能、多功能的RAG检索增强生成系统提供了坚实的基础特别适合需要处理海量数据、支持多模态内容和提供高级查询功能的应用场景。