SpringAI之RAG与文本向量化和向量存储
摘要： 本文介绍了向量数据库的核心概念与应用，重点演示了Redis作为向量存储的实践方案。通过Spring AI和DashScope实现文本向量化（如"text-embedding-v3"模型），将文本转换为浮点数组进行相似性搜索。主要内容包括：Docker部署Redis-Stack、Maven依赖配置、文本嵌入API调用示例，以及向量数据的写入和查询流程。Redis凭借JSO
·
向量与相似度
Spring AI 官网资料
余xian相似度
将文本、图像和视频转换为称为向量（Vectors）的浮点数数组在 VectorStore中，查询与传统关系数据库不同。它们执行相似性搜索，而不是精确匹配。当给定一个向量作为查询时，VectorStore 返回与查询向量“相似”的向量
向量数据库的类型有很多
https://docs.langchain4j.dev/integrations/embedding-stores/
https://docs.spring.io/spring-ai/reference/api/vectordbs/redis.html
使用redis-stack 作为向量数据库演示
docker 安装redis-stack
docker pull docker.1ms.run/redis/redis-stack:latest
maven依赖坐标
<!--spring-ai-alibaba dashscope-->
<dependency>
<groupId>com.alibaba.cloud.ai</groupId>
<artifactId>spring-ai-alibaba-starter-dashscope</artifactId>
</dependency>
<!-- 添加 Redis 向量数据库依赖 -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-redis</artifactId>
</dependency>
文本嵌入模型选择：text-embedding-v3
demo调用实例
import java.util.Arrays;
import java.util.concurrent.Semaphore;
import com.alibaba.dashscope.common.ResultCallback;
import com.alibaba.dashscope.embeddings.TextEmbedding;
import com.alibaba.dashscope.embeddings.TextEmbeddingParam;
import com.alibaba.dashscope.embeddings.TextEmbeddingResult;
import com.alibaba.dashscope.exception.ApiException;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.alibaba.dashscope.utils.Constants;
public final class Main {
public static void main(String[] args) {
try {
TextEmbeddingParam param = TextEmbeddingParam
.builder()
.model("text-embedding-v3")
.texts(Arrays.asList("衣服的质量杠杠的，很漂亮，不枉我等了这么久啊，喜欢，以后还来这里买"))
.build();
TextEmbedding textEmbedding = new TextEmbedding();
TextEmbeddingResult result = textEmbedding.call(param);
System.out.println(result);
} catch (ApiException | NoApiKeyException e) {
System.out.println(e.getMessage());
}
}
}
Redis向量存储相关的配置项
Redis作为向量化存储的特点以及优势：加入了JSON类型
安装命令
docker run -d --name redis-stack-server -p 6379:6379 docker.1ms.run/redis/redis-stack:latest
写入流程
@Autowired VectorStore vectorStore;
// ...
List <Document> documents = List.of(
new Document("Spring AI rocks!! Spring AI rocks!! Spring AI rocks!! Spring AI rocks!! Spring AI rocks!!", Map.of("meta1", "meta1")),
new Document("The World is Big and Salvation Lurks Around the Corner"),
new Document("You walk forward facing the past and you turn back toward the future.", Map.of("meta2", "meta2")));
// Add the documents to Redis
vectorStore.add(documents);
// Retrieve documents similar to a query
List<Document> results = this.vectorStore.similaritySearch(SearchRequest.builder().query("Spring").topK(5).build());
代码演示
package com.atguigu.study.controller;
import com.alibaba.cloud.ai.dashscope.embedding.DashScopeEmbeddingOptions;
import jakarta.annotation.Resource;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.document.Document;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.embedding.EmbeddingOptions;
import org.springframework.ai.embedding.EmbeddingRequest;
import org.springframework.ai.embedding.EmbeddingResponse;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;
@RestController
@Slf4j
public class Embed2VectorController
{
@Resource
private EmbeddingModel embeddingModel;
@Resource
private VectorStore vectorStore;
/**
* 文本向量化
* http://localhost:8011/text2embed?msg=射雕英雄传
*
* @param msg
* @return
*/
@GetMapping("/text2embed")
public EmbeddingResponse text2Embed(String msg)
{
//EmbeddingResponse embeddingResponse = embeddingModel.call(new EmbeddingRequest(List.of(msg), null));
EmbeddingResponse embeddingResponse = embeddingModel.call(new EmbeddingRequest(List.of(msg),
DashScopeEmbeddingOptions.builder()
.withModel("text-embedding-v3")
.build()));
System.out.println(Arrays.toString(embeddingResponse.getResult().getOutput()));
return embeddingResponse;
}
/**
* 文本向量化 后存入向量数据库RedisStack
*/
@GetMapping("/embed2vector/add")
public void add()
{
List<Document> documents = List.of(
new Document("i study LLM"),
new Document("i love java")
);
vectorStore.add(documents);
}
/**
* 从向量数据库RedisStack查找，进行相似度查找
* http://localhost:8011/embed2vector/get?msg=LLM
* @param msg
* @return
*/
@GetMapping("/embed2vector/get")
public List getAll(@RequestParam(name = "msg") String msg)
{
SearchRequest searchRequest = SearchRequest.builder()
.query(msg)
.topK(2)
.build();
List<Document> list = vectorStore.similaritySearch(searchRequest);
System.out.println(list);
return list;
}
}
相关配置
server.port=8011
# 设置响应的字符编码
server.servlet.encoding.charset=utf-8
server.servlet.encoding.enabled=true
server.servlet.encoding.force=true
spring.application.name=SAA-11Embed2vector
aliQwen-api=sk-xxxxxx
# ====SpringAIAlibaba Config=============
spring.ai.dashscope.api-key=${aliQwen-api}
spring.ai.dashscope.chat.options.model=qwen-plus
spring.ai.dashscope.embedding.options.model=text-embedding-v3
# =======Redis Stack==========
spring.data.redis.host=172.18.8.229
spring.data.redis.port=6379
spring.data.redis.username=default
spring.data.redis.password=
spring.ai.vectorstore.redis.initialize-schema=true
spring.ai.vectorstore.redis.index-name=custom-index
spring.ai.vectorstore.redis.prefix=custom-prefix
文本向量化
http://localhost:8011/text2embed?msg=%E5%B0%84%E9%9B%95%E8%8B%B1%E9%9B%84%E4%BC%A0
输出
{
"metadata": {
"model": "text-embedding-v3",
"usage": {
"promptTokens": 0,
"completionTokens": 0,
"totalTokens": 0,
"nativeUsage": {
"total_tokens": 4
}
},
"empty": false
},
"result": {
"index": 0,
"metadata": {
"modalityType": "TEXT",
"documentId": "",
"mimeType": {
"type": "text",
"subtype": "plain",
"parameters": {
},
"charset": null,
"wildcardType": false,
"wildcardSubtype": false,
"subtypeSuffix": null,
"concrete": true
},
"documentData": null
},
"output": [-0.015572611, 0.09280084, -0.067292154, -0.025220128, -0.049362965, -0.045361605, -0.00065707474, 0.04759313, -0.030683527, -0.009089635, 0.027586319, -0.008656794, -0.02593191, -0.019160371, 0.026585978, 0.04374567, 0.08741439, -0.05594213, -0.0039340323, -0.0538645, -0.031991664, -0.035858367, -0.0617518, 0.036897182, -0.025566401, 0.06606096, -0.019362364, -0.04659279, 0.020641645, -0.06336774, -0.0696391, -0.017727192, -0.025412502, -0.022430716, -0.04690059, -0.025585636, 0.0007755045, -0.027778693, -0.09480152, 0.05278721, 0.068330966, -0.005896239, -0.015928501, 0.040552273, -0.025046993, -0.073794365, -0.030356493, -0.036416247, 0.055903655, 0.018717913, -0.016890367, -0.023104023, 0.0080171535, -0.052363988, -0.021103341, 0.04409194, 0.03914795, 0.00423702, 0.022238344, 0.0033521033, 0.0052181236, 0.025181653, 0.02664369, 0.027086148, -0.042168207, 0.091954395, 0.029221492, -0.017323207, -0.006420456, -0.021699699, -0.0063675535, -0.022238344, 0.019141134, 0.018573634, -0.017361682, 0.02512394, -0.008103722, -0.030010222, 0.076718435, 0.02075707, 0.076333694, -0.02083402, 0.026893776, 0.0072043766, -0.00080135465, 0.0014668457, -0.028105726, -0.02614352, 0.0136873545, 0.034877263, -0.0023589765, -0.019477788, 0.0057856245, 0.0026307036, 0.011503918, -0.027624793, -0.08002726, 0.014803119, 0.0062377015, -0.025797248, -0.021045629, 0.040975496, -0.0024527584, -0.040898547, 0.016678758, 0.0071514742, 0.03668557, -0.0068052025, 0.006829249, -0.005064225, 0.06367553, 0.03066429, 0.0102631105, -0.027624793, -0.01307176, -0.034300145, 0.006232892, 0.0035949745, 0.022373004, 0.000853055, -0.0074159876, 0.0018732342, 0.0035709278, -0.041706514, 0.03628159, -0.084413365, 0.044669062, 0.009315673, -0.010272729, 0.018054226, 0.05825061, -0.0033376752, -0.020141475, -0.030818189, -0.00933491, -0.04547703, 0.0068917703, 0.029106067, 0.06940825, -0.0013117448, 0.022353768, -0.0049151354, 0.004359658, -0.028336575, -0.0036166164, -0.041860413, 0.018083083, -0.0006678958, 0.025739536, -0.019814441, 0.01889105, -0.02614352, -0.01127307, 0.015918883, -0.020987917, 0.004244234, -0.06132858, 0.01911228, -0.045592453, 0.048439573, 0.0376282, 0.008021963, -0.055249587, -0.009474381, -0.028105726, 0.036031503, 0.011955995, 0.002678797, -0.008844359, 0.027913352, -0.0013393986, -0.03381921, -0.024431398, -0.03147226, -0.027028436, 0.0077670687, -0.0058866204, 0.009229105, 0.004280304, -0.031645395, -0.026605215, -0.013369938, 0.030914376, -0.0053479755, -0.008599083, 0.018785244, -0.0011331985, 0.025566401, 0.0074015595, 0.04547703, -0.031510733, -0.014610746, 0.033126667, -0.040898547, 0.021026392, -0.022603853, 0.030183358, -0.0035733324, -0.033684548, -0.025450977, -0.03660862, 0.00782959, -0.014947399, 0.044592112, -0.026489792, -0.023392582, -0.05717332, -0.009421478, -0.010282348, -0.08987676, -0.060366716, -0.05040178, 0.048747372, -0.041937362, -0.008425946, -0.018073464, -0.00079113484, -0.029029118, 0.022526903, -0.020545458, -0.018150413, 0.054249246, 0.04532313, 0.049247544, 0.0072139953, -0.020506985, 0.052748736, 0.012648539, -0.025912672, -0.027836405, -0.046361946, 0.012869768, -0.009868746, -0.042552955, 0.046131097, -0.003698375, -0.014668457, -0.021795886, -0.021641986, -0.0045303893, -0.035377435, 0.0023180973, 0.023180973, 0.019631686, -0.03016412, -0.006232892, 0.038128372, -0.024142839, -0.030568104, 0.02175741, 0.016524859, -0.02520089, -0.04928602, 0.008238383, 0.014620365, 0.03224175, 0.01760215, 0.00796906, -0.05467247, -0.006155943, -0.011446206, 0.010465103, 0.003253512, 0.068638764, 0.0023012646, -0.0046073385, 0.047092963, 0.0032150373, 0.008796265, -0.021699699, -0.0035252392, -0.039513458, -0.0005416508, -0.018506303, -0.009426287, -0.016419053, 0.04666974, -0.054787893, -0.008166243, -0.027817167, 0.04174499, 0.051825345, 0.05678857, -0.0064829774, -0.023142498, -0.014024007, -0.0111672655, -0.017159691, 0.011754003, -0.01357193, 0.0014055268, 0.012860149, 0.053018056, 0.0043332065, -0.031510733, 0.05421077, 0.019285414, -0.17621386, -0.034357857, 0.007940205, 0.001229385, -0.015091679, -0.025335552, 0.02924073, 0.0031982046, 0.002197864, 0.0013634452, -0.037936, -0.01759253, 0.009099253, 0.020583933, 0.013533455, 0.03720498, 0.004258662, -0.03635854, 0.0308759, -0.044284314, -0.022584615, -0.018111939, 0.019381601, -0.031491496, 0.0025297077, -0.017361682, 0.0024792098, 0.010782518, -0.010570908, 0.005246979, 0.0062136548, 0.03628159, -0.026220469, 0.011282689, 0.046862114, 0.021699699, -0.00056479574, 0.0012456166, 0.033434466, 0.028413523, -0.004876661, -0.008094103, 0.0050882716, -0.019987578, -0.021911308, -0.026836064, 0.015995832, 0.039436508, 0.0053335475, -0.017852234, -0.041244816, -0.017996514, 0.005646154, -0.01163858, -0.016409434, 0.02987556, -0.0036286397, 0.031183697, 0.0023048716, -0.017409775, 0.009527284, 0.005506683, 0.043399397, 0.024354449, -0.006915817, -0.010782518, -0.014985873, -0.03807066, 0.0062857945, 0.01228303, 0.010503577, 0.031510733, 0.036146928, -0.024373686, 0.009955314, 0.006550308, -0.026585978, 0.028067252, -0.030202594, -0.04890127, 0.004487105, 0.018968, -0.009344528, -0.003784943, -0.006872533, -0.0141394315, 0.007983488, -0.016726851, 0.022238344, 0.25347096, 0.03052963, 0.055480435, -0.026855301, -0.0060164724, -0.03412701, 0.05982807, 0.024873856, -0.029356152, 0.0031188508, -0.03799371, 0.012013707, -0.02391199, 0.0042538526, -0.03381921, -0.0019838489, -0.029490814, 0.017477106, 0.055018738, 0.01760215, 0.0016003046, -0.019410457, 0.0008771016, 0.011176884, 0.0071803303, -0.019756729, -0.018852575, 0.04001363, -0.01228303, 0.05321043, -0.06286757, -0.015966976, -0.0006008657, 0.05859688, 0.03755125, 0.0043548485, -0.0022183037, -0.011513537, -0.018140795, -0.034357857, -0.0693313, -0.026778352, -0.0080412, 0.008166243, -0.031202935, -0.009344528, -0.04767008, -0.037724387, 0.010474722, -0.011898283, 0.002389035, -0.030510392, 0.0036863517, 0.0021629962, 0.027586319, -0.008786647, -0.010493958, 0.018477447, -0.010426628, 0.008594274, 0.046515845, 0.048862796, -0.044284314, -0.0021605918, -0.03985973, 0.036974132, -0.020372324, 0.012898624, 0.021853598, -0.028894456, -0.003253512, 0.009777369, -0.025508689, 0.051979244, -0.0075795045, 0.031780057, 0.0320109, 0.09226219, -0.009512856, 0.0035733324, -0.026316656, -0.047516182, -0.020526221, 0.03174158, -0.012552353, 0.034588702, 0.037532013, 0.02406589, -0.04266838, -0.022392241, 0.007454462, 0.011744385, -0.054249246, 0.01866982, -0.051825345, -0.05344128, -0.011234595, -0.042091258, -0.0025633732, 0.042552955, -0.008021963, -0.005107509, 0.010368916, 0.003763301, 0.031703107, -0.037455063, 0.040744647, 0.0282981, -0.008272048, -0.0028399096, -0.0015017134, -0.012273411, 0.026682165, -0.008421137, 0.048747372, -0.008281667, 0.0049968944, -0.0215458, 0.03441557, 0.015495663, 0.007598742, -0.01643829, 0.0004151053, -0.0018612108, 0.002902431, -0.00065406895, -0.011148028, 0.069831476, -0.04847805, 0.030625816, 0.00869046, -0.02893293, 0.036570147, 0.034588702, 0.07964251, 0.008507705, -0.002765365, 0.018121557, 0.005838527, -0.011330782, -0.00083381764, -0.057596542, -0.03776286, 0.035165824, -0.014716551, -0.0017626196, -0.09618661, -0.02656674, 0.005896239, 0.0068436773, 0.00517003, -0.018438973, 0.008459612, -0.024970043, 0.004280304, 0.010407391, -0.00040007615, -0.028317338, -0.016611427, -0.047631606, 0.031356834, -0.020083763, 0.0323187, 0.09518626, -0.0065839733, 0.07702623, -0.011359639, 0.011965614, 0.036743283, -0.01220608, -0.045515504, -0.034627177, -0.022642327, 0.033511415, 0.057596542, -0.034146246, 0.015178246, -0.003253512, 0.010744044, -0.010388154, -0.0058721923, 0.023642669, 0.057481118, 0.026470555, 0.06067451, 0.023315633, -0.004160071, -0.027047673, -0.02060317, -0.03909024, -0.004107168, 0.13127548, 0.026759114, -0.051325172, 0.067061305, -0.026047332, -0.057750437, 0.045900248, -0.000780915, -0.0020595957, -0.03303048, -0.0143991355, -0.006487787, -0.010667095, -0.02017995, -0.025816485, -0.0062761763, -0.032780394, 0.016197825, -0.06336774, -0.0031188508, -0.009512856, 0.03129912, 0.009132918, 0.018342786, 0.0033881732, 0.022738514, 0.011080697, -0.040783122, -0.020122238, 0.012985192, 0.032280225, -0.0061511337, -0.03978278, -0.011003748, 0.013206421, 0.05609603, 0.035454385, 0.01694808, 0.013773922, 0.0039821253, 0.0099072205, -0.039398033, 0.006665732, 0.017217403, -0.026028097, 0.030356493, -0.00050588144, 0.012417691, -0.03855159, 0.04128329, -0.018111939, -0.0029938081, 0.008517324, -0.004939182, 0.05651925, -0.010186162, 0.01861211, -0.006709016, 0.002599443, -0.0043620626, 0.01673647, -0.02664369, 0.011936758, -0.04259143, 0.028625134, -0.048247203, -0.012398453, -0.047092963, 0.038301505, -0.009825462, 0.029567763, -0.08064285, 0.0045856964, -0.013745066, 0.008512515, 0.009575376, -0.024970043, -0.038743965, -0.02017995, 0.0011812918, 0.008810693, -0.03137607, -0.008276857, 0.00071418553, 0.01967016, 0.022180632, -0.011263452, -0.006872533, 0.04374567, -0.027009198, -0.016496003, -0.002266397, -0.0013033286, -0.018698676, -0.012658157, 0.010897943, 0.031837765, 0.0010604573, 0.007526602, 0.0067282533, -0.029144542, -0.019304652, 0.059481796, 0.026778352, 0.009344528, -0.018362023, -0.015351382, -0.041821938, 0.00077430217, -0.03510811, 0.04136024, -0.025258603, -0.02693225, 0.027451657, 0.01135002, -0.029125305, -0.029740898, 0.0044197747, 0.0024311165, 0.040898547, -0.0029625476, -0.0010183757, -0.009397431, -0.04274533, -0.044207364, -0.013812397, -0.0081277685, -0.04574635, -0.030241068, 0.030933613, -0.021661224, 0.01393744, 0.0013910988, -0.04359177, 0.027086148, -0.005444162, 0.04251448, 0.011888664, -0.009532093, -0.025547164, -0.009349338, 0.0081133405, -0.05586518, 0.02858666, 0.017563675, 0.024296736, 0.0020391562, 0.019583594, 0.023950465, 0.057442643, 0.038955577, -0.023546482, -0.011725147, 0.008454802, -0.025951147, 0.0026379176, 0.032357175, 0.013773922, 0.005083462, 0.0015930907, 0.04540008, -0.018852575, -0.01744825, 0.029971747, -0.025547164, -0.014495322, -0.016726851, -0.014985873, -0.0017193356, -0.022334531, 0.015187865, -0.028394286, -0.0006330281, -0.014639601, -0.004338016, 0.0011217763, -0.0042105685, -0.010763281, -0.005593251, -0.05186382, 0.019064186, -0.03799371, 0.01788109, -0.02929844, 0.007276517, 0.017217403, -0.031337596, 0.0031717534, 0.042476006, -0.012533115, -0.025450977, -0.0018383665, -0.030183358, -0.0010929203, -0.05024788, -0.01701541, 0.02664369, -0.02793259, 0.038436167, -0.020949444, -0.019718254, 0.0014740598, 0.014177905, -0.17298199, 0.0126870135, -0.041167866, 0.02118029, -0.025527926, 0.015957357, 0.0015425927, -0.032992005, 0.03849388, 0.009123299, 0.009522474, 0.031991664, -0.0020187164, -0.008704888, 0.016996173, 0.046092622, 0.038724728, 0.010619001, 0.00463379, -0.0037945616, -0.035858367, 0.0006330281, 0.041937362, 0.038301505, 0.018073464, -0.0195355, 0.0041264053, 0.007536221, -0.032357175, 0.028105726, 0.0040566702, 0.0013514218, 0.0045351987, 0.014447228, 0.026759114, -0.0075795045, -0.0058096712, -0.0013514218, 0.060443662, -0.03418472, -0.016726851, 0.013148709, -0.029144542, -0.008382663, 0.0014440014, 0.048785847, -0.0024960425, -0.0127832005, -0.026412843, -0.041052442, -0.010465103, -0.05867383, 0.024796907, 0.019343127, -0.0024335212, 0.032915056, 0.013610405, -0.009070397, 0.035358198, 0.021141816, 0.005141174, -0.042091258, -0.04151414, 0.025662586, -0.0100515, 0.017775284, 0.027836405, 0.031145222, -0.02183436, -0.03116446, 0.005477827, -0.04439974, 0.0042249965, -0.020218424, 0.001605114, -0.0031789674, 0.051825345, 0.0012323909, -0.024431398, -0.008223955, 0.0034410758, 0.017650243, -0.005444162, 0.059174, 0.028048014, 0.01831393, -0.021161053, 0.003857083, -0.053826027, -0.010484341, -0.002371, -0.064752825, -0.0023734046, 0.012109893, -0.051902294, -0.045592453, -0.015668798, 0.022103682, -0.0031982046, -0.007266898, -0.052594837, -0.017429013, 0.0035733324, -0.020103, -0.041706514, 0.023969702, 0.01271587, 0.015351382, 0.037897523, -0.039590407, 0.02017995, 0.013475744, 0.012196462, 0.003431457, 0.01651524, -0.0073294193, 0.05421077, 0.028048014, -0.019660542, -0.010436247, -0.020699358, 0.040898547, 0.0080412, -0.020968681, -0.01989139, -0.025220128, 0.039051764, 0.008175862, 0.0045568403, -0.007959441, 0.02900988, -0.06575316, 0.039474983, 0.016928842, -0.0019273391, 0.022642327, 0.017929183, 0.025739536, 0.05005551, -0.039128713, 0.013196803, 0.047169913, -0.010032263, -0.015543756, 0.014120194, -0.008983829, -0.015043586, 0.022430716, -0.032915056, 0.0058770017, 0.002649941, 0.013292989, 0.009681182, 0.034357857, -0.02570106, -0.0029649523, 0.02141114, 0.002259183, 0.0008831133, -0.024585297, 0.02118029, 0.0151109155, -0.020276137, 0.01141735, -0.0011710719, -0.024181314, 0.010926799, -0.010109212, 0.016553715, -0.00998417, -0.0061174682, 0.022353768, -0.052171614, -0.01256197, -0.024239024, 0.00790173, -0.0013466125, 0.034588702, 0.0009967337, 0.01860249, 0.013581549, 0.024008177, 0.030683527, -0.007281326, 0.005362403, -0.01400477, 0.029510051, 0.014620365, 0.010118831, -0.02004529, -0.052825686, -0.025104705, -0.00862313, -0.0009522474, -0.0025946337, -0.03720498, -0.015957357, 0.011552012, -0.004424584, 0.006838868, -0.00955614, -0.0069206264, -0.031260647, 0.027086148, 0.0010099594, -0.0055499673, 0.03870549, 0.027316997, -0.035146587, 0.00617518, -0.008906879, -0.042976175, -0.008430756, 0.00042532515, 0.01307176, 0.015553375, 0.026893776, -0.021488087, 0.012860149, -0.0012612469, 0.025027756, 0.019439314, -0.07340962, -0.007060097, 0.026239706, 0.0063819815, -0.04366872, -0.00016351724, 0.0027821977, -0.014456847, 0.0071129994, 0.011157647, 0.0075554578, 0.04955534, -0.022084445, 0.0037368496, 0.051979244, 0.03131836, -0.06132858, -0.0077863056, 0.06286757, 0.0010670702, -0.0014836785, 0.026509028]
},
"results": [
{
"index": 0,
"metadata": {
"modalityType": "TEXT",
"documentId": "",
"mimeType": {
"type": "text",
"subtype": "plain",
"parameters": {
},
"charset": null,
"wildcardType": false,
"wildcardSubtype": false,
"subtypeSuffix": null,
"concrete": true
},
"documentData": null
},
"output": [-0.015572611, 0.09280084, -0.067292154, -0.025220128, -0.049362965, -0.045361605, -0.00065707474, 0.04759313, -0.030683527, -0.009089635, 0.027586319, -0.008656794, -0.02593191, -0.019160371, 0.026585978, 0.04374567, 0.08741439, -0.05594213, -0.0039340323, -0.0538645, -0.031991664, -0.035858367, -0.0617518, 0.036897182, -0.025566401, 0.06606096, -0.019362364, -0.04659279, 0.020641645, -0.06336774, -0.0696391, -0.017727192, -0.025412502, -0.022430716, -0.04690059, -0.025585636, 0.0007755045, -0.027778693, -0.09480152, 0.05278721, 0.068330966, -0.005896239, -0.015928501, 0.040552273, -0.025046993, -0.073794365, -0.030356493, -0.036416247, 0.055903655, 0.018717913, -0.016890367, -0.023104023, 0.0080171535, -0.052363988, -0.021103341, 0.04409194, 0.03914795, 0.00423702, 0.022238344, 0.0033521033, 0.0052181236, 0.025181653, 0.02664369, 0.027086148, -0.042168207, 0.091954395, 0.029221492, -0.017323207, -0.006420456, -0.021699699, -0.0063675535, -0.022238344, 0.019141134, 0.018573634, -0.017361682, 0.02512394, -0.008103722, -0.030010222, 0.076718435, 0.02075707, 0.076333694, -0.02083402, 0.026893776, 0.0072043766, -0.00080135465, 0.0014668457, -0.028105726, -0.02614352, 0.0136873545, 0.034877263, -0.0023589765, -0.019477788, 0.0057856245, 0.0026307036, 0.011503918, -0.027624793, -0.08002726, 0.014803119, 0.0062377015, -0.025797248, -0.021045629, 0.040975496, -0.0024527584, -0.040898547, 0.016678758, 0.0071514742, 0.03668557, -0.0068052025, 0.006829249, -0.005064225, 0.06367553, 0.03066429, 0.0102631105, -0.027624793, -0.01307176, -0.034300145, 0.006232892, 0.0035949745, 0.022373004, 0.000853055, -0.0074159876, 0.0018732342, 0.0035709278, -0.041706514, 0.03628159, -0.084413365, 0.044669062, 0.009315673, -0.010272729, 0.018054226, 0.05825061, -0.0033376752, -0.020141475, -0.030818189, -0.00933491, -0.04547703, 0.0068917703, 0.029106067, 0.06940825, -0.0013117448, 0.022353768, -0.0049151354, 0.004359658, -0.028336575, -0.0036166164, -0.041860413, 0.018083083, -0.0006678958, 0.025739536, -0.019814441, 0.01889105, -0.02614352, -0.01127307, 0.015918883, -0.020987917, 0.004244234, -0.06132858, 0.01911228, -0.045592453, 0.048439573, 0.0376282, 0.008021963, -0.055249587, -0.009474381, -0.028105726, 0.036031503, 0.011955995, 0.002678797, -0.008844359, 0.027913352, -0.0013393986, -0.03381921, -0.024431398, -0.03147226, -0.027028436, 0.0077670687, -0.0058866204, 0.009229105, 0.004280304, -0.031645395, -0.026605215, -0.013369938, 0.030914376, -0.0053479755, -0.008599083, 0.018785244, -0.0011331985, 0.025566401, 0.0074015595, 0.04547703, -0.031510733, -0.014610746, 0.033126667, -0.040898547, 0.021026392, -0.022603853, 0.030183358, -0.0035733324, -0.033684548, -0.025450977, -0.03660862, 0.00782959, -0.014947399, 0.044592112, -0.026489792, -0.023392582, -0.05717332, -0.009421478, -0.010282348, -0.08987676, -0.060366716, -0.05040178, 0.048747372, -0.041937362, -0.008425946, -0.018073464, -0.00079113484, -0.029029118, 0.022526903, -0.020545458, -0.018150413, 0.054249246, 0.04532313, 0.049247544, 0.0072139953, -0.020506985, 0.052748736, 0.012648539, -0.025912672, -0.027836405, -0.046361946, 0.012869768, -0.009868746, -0.042552955, 0.046131097, -0.003698375, -0.014668457, -0.021795886, -0.021641986, -0.0045303893, -0.035377435, 0.0023180973, 0.023180973, 0.019631686, -0.03016412, -0.006232892, 0.038128372, -0.024142839, -0.030568104, 0.02175741, 0.016524859, -0.02520089, -0.04928602, 0.008238383, 0.014620365, 0.03224175, 0.01760215, 0.00796906, -0.05467247, -0.006155943, -0.011446206, 0.010465103, 0.003253512, 0.068638764, 0.0023012646, -0.0046073385, 0.047092963, 0.0032150373, 0.008796265, -0.021699699, -0.0035252392, -0.039513458, -0.0005416508, -0.018506303, -0.009426287, -0.016419053, 0.04666974, -0.054787893, -0.008166243, -0.027817167, 0.04174499, 0.051825345, 0.05678857, -0.0064829774, -0.023142498, -0.014024007, -0.0111672655, -0.017159691, 0.011754003, -0.01357193, 0.0014055268, 0.012860149, 0.053018056, 0.0043332065, -0.031510733, 0.05421077, 0.019285414, -0.17621386, -0.034357857, 0.007940205, 0.001229385, -0.015091679, -0.025335552, 0.02924073, 0.0031982046, 0.002197864, 0.0013634452, -0.037936, -0.01759253, 0.009099253, 0.020583933, 0.013533455, 0.03720498, 0.004258662, -0.03635854, 0.0308759, -0.044284314, -0.022584615, -0.018111939, 0.019381601, -0.031491496, 0.0025297077, -0.017361682, 0.0024792098, 0.010782518, -0.010570908, 0.005246979, 0.0062136548, 0.03628159, -0.026220469, 0.011282689, 0.046862114, 0.021699699, -0.00056479574, 0.0012456166, 0.033434466, 0.028413523, -0.004876661, -0.008094103, 0.0050882716, -0.019987578, -0.021911308, -0.026836064, 0.015995832, 0.039436508, 0.0053335475, -0.017852234, -0.041244816, -0.017996514, 0.005646154, -0.01163858, -0.016409434, 0.02987556, -0.0036286397, 0.031183697, 0.0023048716, -0.017409775, 0.009527284, 0.005506683, 0.043399397, 0.024354449, -0.006915817, -0.010782518, -0.014985873, -0.03807066, 0.0062857945, 0.01228303, 0.010503577, 0.031510733, 0.036146928, -0.024373686, 0.009955314, 0.006550308, -0.026585978, 0.028067252, -0.030202594, -0.04890127, 0.004487105, 0.018968, -0.009344528, -0.003784943, -0.006872533, -0.0141394315, 0.007983488, -0.016726851, 0.022238344, 0.25347096, 0.03052963, 0.055480435, -0.026855301, -0.0060164724, -0.03412701, 0.05982807, 0.024873856, -0.029356152, 0.0031188508, -0.03799371, 0.012013707, -0.02391199, 0.0042538526, -0.03381921, -0.0019838489, -0.029490814, 0.017477106, 0.055018738, 0.01760215, 0.0016003046, -0.019410457, 0.0008771016, 0.011176884, 0.0071803303, -0.019756729, -0.018852575, 0.04001363, -0.01228303, 0.05321043, -0.06286757, -0.015966976, -0.0006008657, 0.05859688, 0.03755125, 0.0043548485, -0.0022183037, -0.011513537, -0.018140795, -0.034357857, -0.0693313, -0.026778352, -0.0080412, 0.008166243, -0.031202935, -0.009344528, -0.04767008, -0.037724387, 0.010474722, -0.011898283, 0.002389035, -0.030510392, 0.0036863517, 0.0021629962, 0.027586319, -0.008786647, -0.010493958, 0.018477447, -0.010426628, 0.008594274, 0.046515845, 0.048862796, -0.044284314, -0.0021605918, -0.03985973, 0.036974132, -0.020372324, 0.012898624, 0.021853598, -0.028894456, -0.003253512, 0.009777369, -0.025508689, 0.051979244, -0.0075795045, 0.031780057, 0.0320109, 0.09226219, -0.009512856, 0.0035733324, -0.026316656, -0.047516182, -0.020526221, 0.03174158, -0.012552353, 0.034588702, 0.037532013, 0.02406589, -0.04266838, -0.022392241, 0.007454462, 0.011744385, -0.054249246, 0.01866982, -0.051825345, -0.05344128, -0.011234595, -0.042091258, -0.0025633732, 0.042552955, -0.008021963, -0.005107509, 0.010368916, 0.003763301, 0.031703107, -0.037455063, 0.040744647, 0.0282981, -0.008272048, -0.0028399096, -0.0015017134, -0.012273411, 0.026682165, -0.008421137, 0.048747372, -0.008281667, 0.0049968944, -0.0215458, 0.03441557, 0.015495663, 0.007598742, -0.01643829, 0.0004151053, -0.0018612108, 0.002902431, -0.00065406895, -0.011148028, 0.069831476, -0.04847805, 0.030625816, 0.00869046, -0.02893293, 0.036570147, 0.034588702, 0.07964251, 0.008507705, -0.002765365, 0.018121557, 0.005838527, -0.011330782, -0.00083381764, -0.057596542, -0.03776286, 0.035165824, -0.014716551, -0.0017626196, -0.09618661, -0.02656674, 0.005896239, 0.0068436773, 0.00517003, -0.018438973, 0.008459612, -0.024970043, 0.004280304, 0.010407391, -0.00040007615, -0.028317338, -0.016611427, -0.047631606, 0.031356834, -0.020083763, 0.0323187, 0.09518626, -0.0065839733, 0.07702623, -0.011359639, 0.011965614, 0.036743283, -0.01220608, -0.045515504, -0.034627177, -0.022642327, 0.033511415, 0.057596542, -0.034146246, 0.015178246, -0.003253512, 0.010744044, -0.010388154, -0.0058721923, 0.023642669, 0.057481118, 0.026470555, 0.06067451, 0.023315633, -0.004160071, -0.027047673, -0.02060317, -0.03909024, -0.004107168, 0.13127548, 0.026759114, -0.051325172, 0.067061305, -0.026047332, -0.057750437, 0.045900248, -0.000780915, -0.0020595957, -0.03303048, -0.0143991355, -0.006487787, -0.010667095, -0.02017995, -0.025816485, -0.0062761763, -0.032780394, 0.016197825, -0.06336774, -0.0031188508, -0.009512856, 0.03129912, 0.009132918, 0.018342786, 0.0033881732, 0.022738514, 0.011080697, -0.040783122, -0.020122238, 0.012985192, 0.032280225, -0.0061511337, -0.03978278, -0.011003748, 0.013206421, 0.05609603, 0.035454385, 0.01694808, 0.013773922, 0.0039821253, 0.0099072205, -0.039398033, 0.006665732, 0.017217403, -0.026028097, 0.030356493, -0.00050588144, 0.012417691, -0.03855159, 0.04128329, -0.018111939, -0.0029938081, 0.008517324, -0.004939182, 0.05651925, -0.010186162, 0.01861211, -0.006709016, 0.002599443, -0.0043620626, 0.01673647, -0.02664369, 0.011936758, -0.04259143, 0.028625134, -0.048247203, -0.012398453, -0.047092963, 0.038301505, -0.009825462, 0.029567763, -0.08064285, 0.0045856964, -0.013745066, 0.008512515, 0.009575376, -0.024970043, -0.038743965, -0.02017995, 0.0011812918, 0.008810693, -0.03137607, -0.008276857, 0.00071418553, 0.01967016, 0.022180632, -0.011263452, -0.006872533, 0.04374567, -0.027009198, -0.016496003, -0.002266397, -0.0013033286, -0.018698676, -0.012658157, 0.010897943, 0.031837765, 0.0010604573, 0.007526602, 0.0067282533, -0.029144542, -0.019304652, 0.059481796, 0.026778352, 0.009344528, -0.018362023, -0.015351382, -0.041821938, 0.00077430217, -0.03510811, 0.04136024, -0.025258603, -0.02693225, 0.027451657, 0.01135002, -0.029125305, -0.029740898, 0.0044197747, 0.0024311165, 0.040898547, -0.0029625476, -0.0010183757, -0.009397431, -0.04274533, -0.044207364, -0.013812397, -0.0081277685, -0.04574635, -0.030241068, 0.030933613, -0.021661224, 0.01393744, 0.0013910988, -0.04359177, 0.027086148, -0.005444162, 0.04251448, 0.011888664, -0.009532093, -0.025547164, -0.009349338, 0.0081133405, -0.05586518, 0.02858666, 0.017563675, 0.024296736, 0.0020391562, 0.019583594, 0.023950465, 0.057442643, 0.038955577, -0.023546482, -0.011725147, 0.008454802, -0.025951147, 0.0026379176, 0.032357175, 0.013773922, 0.005083462, 0.0015930907, 0.04540008, -0.018852575, -0.01744825, 0.029971747, -0.025547164, -0.014495322, -0.016726851, -0.014985873, -0.0017193356, -0.022334531, 0.015187865, -0.028394286, -0.0006330281, -0.014639601, -0.004338016, 0.0011217763, -0.0042105685, -0.010763281, -0.005593251, -0.05186382, 0.019064186, -0.03799371, 0.01788109, -0.02929844, 0.007276517, 0.017217403, -0.031337596, 0.0031717534, 0.042476006, -0.012533115, -0.025450977, -0.0018383665, -0.030183358, -0.0010929203, -0.05024788, -0.01701541, 0.02664369, -0.02793259, 0.038436167, -0.020949444, -0.019718254, 0.0014740598, 0.014177905, -0.17298199, 0.0126870135, -0.041167866, 0.02118029, -0.025527926, 0.015957357, 0.0015425927, -0.032992005, 0.03849388, 0.009123299, 0.009522474, 0.031991664, -0.0020187164, -0.008704888, 0.016996173, 0.046092622, 0.038724728, 0.010619001, 0.00463379, -0.0037945616, -0.035858367, 0.0006330281, 0.041937362, 0.038301505, 0.018073464, -0.0195355, 0.0041264053, 0.007536221, -0.032357175, 0.028105726, 0.0040566702, 0.0013514218, 0.0045351987, 0.014447228, 0.026759114, -0.0075795045, -0.0058096712, -0.0013514218, 0.060443662, -0.03418472, -0.016726851, 0.013148709, -0.029144542, -0.008382663, 0.0014440014, 0.048785847, -0.0024960425, -0.0127832005, -0.026412843, -0.041052442, -0.010465103, -0.05867383, 0.024796907, 0.019343127, -0.0024335212, 0.032915056, 0.013610405, -0.009070397, 0.035358198, 0.021141816, 0.005141174, -0.042091258, -0.04151414, 0.025662586, -0.0100515, 0.017775284, 0.027836405, 0.031145222, -0.02183436, -0.03116446, 0.005477827, -0.04439974, 0.0042249965, -0.020218424, 0.001605114, -0.0031789674, 0.051825345, 0.0012323909, -0.024431398, -0.008223955, 0.0034410758, 0.017650243, -0.005444162, 0.059174, 0.028048014, 0.01831393, -0.021161053, 0.003857083, -0.053826027, -0.010484341, -0.002371, -0.064752825, -0.0023734046, 0.012109893, -0.051902294, -0.045592453, -0.015668798, 0.022103682, -0.0031982046, -0.007266898, -0.052594837, -0.017429013, 0.0035733324, -0.020103, -0.041706514, 0.023969702, 0.01271587, 0.015351382, 0.037897523, -0.039590407, 0.02017995, 0.013475744, 0.012196462, 0.003431457, 0.01651524, -0.0073294193, 0.05421077, 0.028048014, -0.019660542, -0.010436247, -0.020699358, 0.040898547, 0.0080412, -0.020968681, -0.01989139, -0.025220128, 0.039051764, 0.008175862, 0.0045568403, -0.007959441, 0.02900988, -0.06575316, 0.039474983, 0.016928842, -0.0019273391, 0.022642327, 0.017929183, 0.025739536, 0.05005551, -0.039128713, 0.013196803, 0.047169913, -0.010032263, -0.015543756, 0.014120194, -0.008983829, -0.015043586, 0.022430716, -0.032915056, 0.0058770017, 0.002649941, 0.013292989, 0.009681182, 0.034357857, -0.02570106, -0.0029649523, 0.02141114, 0.002259183, 0.0008831133, -0.024585297, 0.02118029, 0.0151109155, -0.020276137, 0.01141735, -0.0011710719, -0.024181314, 0.010926799, -0.010109212, 0.016553715, -0.00998417, -0.0061174682, 0.022353768, -0.052171614, -0.01256197, -0.024239024, 0.00790173, -0.0013466125, 0.034588702, 0.0009967337, 0.01860249, 0.013581549, 0.024008177, 0.030683527, -0.007281326, 0.005362403, -0.01400477, 0.029510051, 0.014620365, 0.010118831, -0.02004529, -0.052825686, -0.025104705, -0.00862313, -0.0009522474, -0.0025946337, -0.03720498, -0.015957357, 0.011552012, -0.004424584, 0.006838868, -0.00955614, -0.0069206264, -0.031260647, 0.027086148, 0.0010099594, -0.0055499673, 0.03870549, 0.027316997, -0.035146587, 0.00617518, -0.008906879, -0.042976175, -0.008430756, 0.00042532515, 0.01307176, 0.015553375, 0.026893776, -0.021488087, 0.012860149, -0.0012612469, 0.025027756, 0.019439314, -0.07340962, -0.007060097, 0.026239706, 0.0063819815, -0.04366872, -0.00016351724, 0.0027821977, -0.014456847, 0.0071129994, 0.011157647, 0.0075554578, 0.04955534, -0.022084445, 0.0037368496, 0.051979244, 0.03131836, -0.06132858, -0.0077863056, 0.06286757, 0.0010670702, -0.0014836785, 0.026509028]
}
]
}
存入redis中
相似度查找
从向量数据库RedisStack查找，进行相似度查找
http://localhost:8011/embed2vector/get?msg=LLM
知识图谱
检索增强生成RAG
RAG 流程分为两个不同的阶段：索引和检索
Index
检索Retrieval
知识库问答系统
知识库解析与存储
package com.atguigu.study.config;
import cn.hutool.crypto.SecureUtil;
import jakarta.annotation.PostConstruct;
import org.springframework.ai.document.Document;
import org.springframework.ai.reader.TextReader;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.AbstractVectorStoreBuilder;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;
import org.springframework.data.redis.core.RedisTemplate;
import java.nio.charset.Charset;
import java.util.List;
@Configuration
public class InitVectorDatabaseConfig
{
@Autowired
private VectorStore vectorStore;
@Autowired
private RedisTemplate<String,String> redisTemplate;
@Value("classpath:ops.txt")
private Resource opsFile;
@PostConstruct
public void init()
{
//1 读取文件
TextReader textReader = new TextReader(opsFile);
textReader.setCharset(Charset.defaultCharset());
//2 文件转换为向量(开启分词)
List<Document> list = new TokenTextSplitter().transform(textReader.read());
//3 写入向量数据库RedisStack
//vectorStore.add(list);
// 解决上面第3步，向量数据重复问题，使用redis setnx命令处理
//4 去重复版本
String sourceMetadata = (String)textReader.getCustomMetadata().get("source");
String textHash = SecureUtil.md5(sourceMetadata);
String redisKey = "vector-xxx:" + textHash;
// 判断是否存入过,redisKey如果可以成功插入表示以前没有过，可以假如向量数据
Boolean retFlag = redisTemplate.opsForValue().setIfAbsent(redisKey, "1");
System.out.println("****retFlag : "+retFlag);
if(Boolean.TRUE.equals(retFlag))
{
//键不存在，首次插入,可以保存进向量数据库
vectorStore.add(list);
}else {
//键已存在，跳过或者报错
//throw new RuntimeException("---重复操作");
System.out.println("------向量初始化数据已经加载过，请不要重复操作");
}
}
}
大模型客户端配置
package com.atguigu.study.config;
import com.alibaba.cloud.ai.dashscope.api.DashScopeApi;
import com.alibaba.cloud.ai.dashscope.chat.DashScopeChatModel;
import com.alibaba.cloud.ai.dashscope.chat.DashScopeChatOptions;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.ChatOptions;
import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor;
import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class SaaLLMConfig
{
// 模型名称常量定义
private final String DEEPSEEK_MODEL = "deepseek-v3";
private final String QWEN_MODEL = "qwen-plus";
@Bean(name = "deepseek")
public ChatModel deepSeek()
{
return DashScopeChatModel.builder()
.dashScopeApi(DashScopeApi.builder()
.apiKey(System.getProperty("aliQwen-api"))
.build())
.defaultOptions(
DashScopeChatOptions.builder().withModel(DEEPSEEK_MODEL).build()
)
.build();
}
@Bean(name = "qwen")
public ChatModel qwen()
{
return DashScopeChatModel.builder().dashScopeApi(DashScopeApi.builder()
.apiKey(System.getProperty("aliQwen-api"))
.build())
.defaultOptions(
DashScopeChatOptions.builder()
.withModel(QWEN_MODEL)
.build()
)
.build();
}
@Bean(name = "deepseekChatClient")
public ChatClient deepseekChatClient(@Qualifier("deepseek") ChatModel deepSeek)
{
return ChatClient.builder(deepSeek)
.defaultOptions(ChatOptions.builder()
.model(DEEPSEEK_MODEL)
.build())
.build();
}
@Bean(name = "qwenChatClient")
public ChatClient qwenChatClient(@Qualifier("qwen") ChatModel qwen)
{
return ChatClient.builder(qwen)
.defaultOptions(ChatOptions.builder()
.model(QWEN_MODEL)
.build())
.build();
}
}
问答助手
package com.atguigu.study.controller;
import jakarta.annotation.Resource;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor;
import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;
@RestController
public class RagController
{
@Resource(name = "qwenChatClient")
private ChatClient chatClient;
@Resource
private VectorStore vectorStore;
/**
* http://localhost:8012/rag4aiops?msg=00000
* http://localhost:8012/rag4aiops?msg=C2222
* @param msg
* @return
*/
@GetMapping("/rag4aiops")
public Flux<String> rag(String msg)
{
String systemInfo = """
你是一个知识库检索工程师,按照给出的关键词给出对应名词解释,否则回复找不到信息。
""";
RetrievalAugmentationAdvisor advisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(vectorStore).build())
.build();
return chatClient
.prompt()
.system(systemInfo)
.user(msg)
.advisors(advisor)
.stream()
.content();
}
}
演示效果
知识库信息
SCA 自动识别应用程序中使用的第三方开源组件（如 Maven、npm、PyPI 依赖），并检测其中已知的安全漏洞、许可证风险和版本过期问题。
SAST 在不运行程序的情况下，通过分析源代码、字节码或二进制文件，检测潜在安全缺陷（如 SQL 注入、XSS、缓冲区溢出）。
DAST 在应用程序运行时，通过模拟外部攻击（如发送恶意 HTTP 请求），从黑盒角度检测安全漏洞。
IAST 结合 SAST（白盒） + DAST（黑盒） 的混合方案，在应用运行时通过 Agent/探针 监控代码执行路径与数据流，实现高精度漏洞检测。
RASP 运行时应用自我保护
RocketMQ RocketMQ消息队列
Redis 非关系型数据库
MySQL 关系型数据库
CRM 客户关系系统
MOOOO 请求成功
C1111 请求失败
AVS 病毒检测
TokenTextSplitter的分析器的默认配置：
public class TokenTextSplitter extends TextSplitter {
private static final int DEFAULT_CHUNK_SIZE = 800;
private static final int MIN_CHUNK_SIZE_CHARS = 350;
private static final int MIN_CHUNK_LENGTH_TO_EMBED = 5;
private static final int MAX_NUM_CHUNKS = 10000;
private static final boolean KEEP_SEPARATOR = true;
private final EncodingRegistry registry;
private final Encoding encoding;
private final int chunkSize;
private final int minChunkSizeChars;
private final int minChunkLengthToEmbed;
private final int maxNumChunks;
private final boolean keepSeparator;
public TokenTextSplitter() {
this(800, 350, 5, 10000, true);
}
RAG的执行逻辑
原始文本
↓
[Text Splitter] → 将长文本切分为小块（chunks）
↓
[Embedding Model] → 为每个 chunk 生成向量（embedding）
↓
[Vector Store] → 存储 (chunk, embedding) 对
↓
查询时：用相同 embedding 模型编码 query → 向量相似性搜索 → 返回相关 chunks
RedisVendor中doAdd函数中已经封装了相关的Emdedding Model的信息
同时阿里百炼平台封装DashScopeCloudStore的add方法
//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by FernFlower decompiler)
//
package com.alibaba.cloud.ai.dashscope.rag;
import com.alibaba.cloud.ai.dashscope.api.DashScopeApi;
import com.alibaba.cloud.ai.dashscope.common.DashScopeException;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.Filter;
import org.springframework.util.Assert;
import org.springframework.util.CollectionUtils;
public class DashScopeCloudStore implements VectorStore {
private final DashScopeStoreOptions options;
private final DashScopeApi dashScopeApi;
public DashScopeCloudStore(DashScopeApi dashScopeApi, DashScopeStoreOptions options) {
Assert.notNull(options, "DashScopeStoreOptions must not be null");
Assert.notNull(options.getIndexName(), "IndexName must not be null");
this.options = options;
this.dashScopeApi = dashScopeApi;
}
public String getName() {
return super.getName();
}
public void add(List<Document> documents) {
if (CollectionUtils.isEmpty(documents)) {
throw new DashScopeException("documents must not be null");
} else {
List<String> documentIdList = (List)documents.stream().filter((e) -> {
return e.getId() != null;
}).map(Document::getId).collect(Collectors.toList());
if (documentIdList != null && !documentIdList.isEmpty()) {
this.dashScopeApi.upsertPipeline(documents, this.options);
} else {
throw new DashScopeException("document's id must not be null");
}
}
}
public void delete(List<String> idList) {
String pipelineId = this.dashScopeApi.getPipelineIdByName(this.options.getIndexName());
if (pipelineId == null) {
throw new DashScopeException("Index:" + this.options.getIndexName() + " NotExist");
} else {
this.dashScopeApi.deletePipelineDocument(pipelineId, idList);
}
}
public void delete(Filter.Expression filterExpression) {
}
public List<Document> similaritySearch(String query) {
return this.similaritySearch(SearchRequest.builder().query(query).build());
}
public <T> Optional<T> getNativeClient() {
return super.getNativeClient();
}
public List<Document> similaritySearch(SearchRequest request) {
String pipelineId = this.dashScopeApi.getPipelineIdByName(this.options.getIndexName());
if (pipelineId == null) {
throw new DashScopeException("Index:" + this.options.getIndexName() + " NotExist");
} else {
DashScopeDocumentRetrieverOptions searchOption = this.options.getRetrieverOptions();
if (searchOption == null) {
searchOption = new DashScopeDocumentRetrieverOptions();
}
searchOption.setRerankTopN(request.getTopK());
return this.dashScopeApi.retriever(pipelineId, request.getQuery(), searchOption);
}
}
}
配置信息
server.port=8012
# 设置全局编码格式
server.servlet.encoding.enabled=true
server.servlet.encoding.force=true
server.servlet.encoding.charset=UTF-8
spring.application.name=SAA-12RAG4AiDatabase
aliQwen-api=sk-xxxx
# ====SpringAIAlibaba Config=============
spring.ai.dashscope.api-key=${aliQwen-api}
spring.ai.dashscope.chat.options.model=deepseek-r1
spring.ai.dashscope.embedding.options.model=text-embedding-v3
# =======Redis Stack==========
spring.data.redis.host=172.18.8.229
spring.data.redis.port=6379
spring.data.redis.username=default
spring.data.redis.password=
spring.ai.vectorstore.redis.initialize-schema=true
spring.ai.vectorstore.redis.index-name=sssc-index
spring.ai.vectorstore.redis.prefix=sssc-prefix
其他相关知识
嵌入(Embedding)
嵌入(Embedding)的工作原理是将文本、图像和视频转换为称为向量（Vectors）的浮点数数组。这些向量旨在捕捉文本、图像和视频的含义。嵌入数组的长度称为向量的维度（Dimensionality）。
嵌入模型（EmbeddingModel）
嵌入模型（EmbeddingModel）是嵌入过程中采用的模型。当前EmbeddingModel的接口主要用于将文本转换为数值向量，接口的设计主要围绕这两个目标展开：
- 可移植性：该接口确保在各种嵌入模型之间的轻松适配。它允许开发者在不同的嵌入技术或模型之间切换，所需的代码更改最小化。这一设计与 Spring 模块化和互换性的理念一致。
- 简单性：嵌入模型简化了文本转换为嵌入的过程。通过提供如embed(String text)和embed(Document document)这样简单的方法，它去除了处理原始文本数据和嵌入算法的复杂性。这个设计选择使开发者，尤其是那些初次接触 AI 的开发者，更容易在他们的应用程序中使用嵌入，而无需深入了解其底层机制。
EmbeddingModel API 接口
/*
* Copyright 2023-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.ai.embedding;
import java.util.ArrayList;
import java.util.List;
import org.springframework.ai.document.Document;
import org.springframework.ai.model.Model;
import org.springframework.util.Assert;
/**
* EmbeddingModel is a generic interface for embedding models.
*
* @author Mark Pollack
* @author Christian Tzolov
* @author Josh Long
* @author Soby Chacko
* @author Jihoon Kim
* @since 1.0.0
*
*/
public interface EmbeddingModel extends Model<EmbeddingRequest, EmbeddingResponse> {
@Override
EmbeddingResponse call(EmbeddingRequest request);
/**
* Embeds the given text into a vector.
* @param text the text to embed.
* @return the embedded vector.
*/
default float[] embed(String text) {
Assert.notNull(text, "Text must not be null");
List<float[]> response = this.embed(List.of(text));
return response.iterator().next();
}
/**
* Embeds the given document's content into a vector.
* @param document the document to embed.
* @return the embedded vector.
*/
float[] embed(Document document);
/**
* Embeds a batch of texts into vectors.
* @param texts list of texts to embed.
* @return list of embedded vectors.
*/
default List<float[]> embed(List<String> texts) {
Assert.notNull(texts, "Texts must not be null");
return this.call(new EmbeddingRequest(texts, EmbeddingOptionsBuilder.builder().build()))
.getResults()
.stream()
.map(Embedding::getOutput)
.toList();
}
/**
* Embeds a batch of {@link Document}s into vectors based on a
* {@link BatchingStrategy}.
* @param documents list of {@link Document}s.
* @param options {@link EmbeddingOptions}.
* @param batchingStrategy {@link BatchingStrategy}.
* @return a list of float[] that represents the vectors for the incoming
* {@link Document}s. The returned list is expected to be in the same order of the
* {@link Document} list.
*/
default List<float[]> embed(List<Document> documents, EmbeddingOptions options, BatchingStrategy batchingStrategy) {
Assert.notNull(documents, "Documents must not be null");
List<float[]> embeddings = new ArrayList<>(documents.size());
List<List<Document>> batch = batchingStrategy.batch(documents);
for (List<Document> subBatch : batch) {
List<String> texts = subBatch.stream().map(Document::getText).toList();
EmbeddingRequest request = new EmbeddingRequest(texts, options);
EmbeddingResponse response = this.call(request);
for (int i = 0; i < subBatch.size(); i++) {
embeddings.add(response.getResults().get(i).getOutput());
}
}
Assert.isTrue(embeddings.size() == documents.size(),
"Embeddings must have the same number as that of the documents");
return embeddings;
}
/**
* Embeds a batch of texts into vectors and returns the {@link EmbeddingResponse}.
* @param texts list of texts to embed.
* @return the embedding response.
*/
default EmbeddingResponse embedForResponse(List<String> texts) {
Assert.notNull(texts, "Texts must not be null");
return this.call(new EmbeddingRequest(texts, EmbeddingOptionsBuilder.builder().build()));
}
/**
* Get the number of dimensions of the embedded vectors. Note that by default, this
* method will call the remote Embedding endpoint to get the dimensions of the
* embedded vectors. If the dimensions are known ahead of time, it is recommended to
* override this method.
* @return the number of dimensions of the embedded vectors.
*/
default int dimensions() {
return embed("Test String").length;
}
}
Embedding Model API提供多种选项，将文本转换为Embeddings，支持单个字符串、结构化的Document对象或文本批处理。
有多种快捷方式可以获得文本Embeddings。例如embed(String text)方法，它接受单个字符串并返回相应的 Embedding 向量。所有方法都围绕着call方法实现，这是调用 Embedding Model的主要方法。
通常，Embedding返回一个float数组，以数值向量格式表示Embeddings。
embedForResponse方法提供了更全面的输出，可能包括有关Embeddings的其他信息。
dimensions方法是开发人员快速确定 Embedding 向量大小的便利工具，这对于理解 Embedding space 和后续处理步骤非常重要。
EmbeddingRequest
EmbeddingRequest是一种ModelRequest，它接受文本对象列表和可选的Embedding请求选项。以下代码片段简要地显示了 EmbeddingRequest 类，省略了构造函数和其他工具方法：
public class EmbeddingRequest implements ModelRequest<List<String>> {
private final List<String> inputs;
private final EmbeddingOptions options;
// other methods omitted
}
EmbeddingResponse
EmbeddingResponse类的结构如下：
public class EmbeddingResponse implements ModelResponse<Embedding> {
private List<Embedding> embeddings;
private EmbeddingResponseMetadata metadata = new EmbeddingResponseMetadata();
// other methods omitted
}
Embedding
Embedding表示一个 Embedding 向量。
public class Embedding implements ModelResult<List<Double>> {
private List<Double> embedding;
private Integer index;
private EmbeddingResultMetadata metadata;
// other methods omitted
}
向量存储
向量存储（VectorStore）是一种用于存储和检索高维向量数据的数据库或存储解决方案，它特别适用于处理那些经过嵌入模型转化后的数据。在 VectorStore 中，查询与传统关系数据库不同。它们执行相似性搜索，而不是精确匹配。当给定一个向量作为查询时，VectorStore 返回与查询向量“相似”的向量。
VectorStore 用于将您的数据与 AI 模型集成。在使用它们时的第一步是将您的数据加载到矢量数据库中。然后，当要将用户查询发送到 AI 模型时，首先检索一组相似文档。然后，这些文档作为用户问题的上下文，并与用户的查询一起发送到 AI 模型。这种技术被称为检索增强生成（Retrieval Augmented Generation，RAG）。
Spring AI 提供了一个抽象化的 API，通过 VectorStore 接口与向量数据库进行交互。
有“AI”的1024 = 2048，欢迎大家加入2048 AI社区
更多推荐
7
0- 0
已为社区贡献3条内容
所有评论(0)