以Word2Vec 词向量转化三国人物，找到相似角色

一，应用场景​

​行业痛点​：

• 传统推荐系统依赖人工规则，难以捕捉语义关联
• 非结构化数据（如酒店描述、用户评论）利用率低

​解决方案​：

• Embedding技术将文本/图像转化为向量
• 向量数据库实现级相似性检索

 

二、技术解析（核心模块）​​

1. ​Embedding技术三重奏​

• ​N-Gram特征工程​

  # 提取酒店描述的Bi-Gram特征
  vec = CountVectorizer(ngram_range=(2,2))
  bigram_features = vec.fit_transform(df['desc'])

  ​作用​：捕捉"免费WiFi"、"海景房"等组合特征

• ​TF-IDF加权策略​
  TFIDF=文档总词数词频​×log包含该词的文档数文档总数​
  ​优势​：降低"酒店"、"服务"等高频词的权重

• ​Word2Vec深度语义编码​

  ​案例​：king - man + woman ≈ queen

2. ​余弦相似度实战​

# 计算酒店相似度矩阵
cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)

​公式​：
cos(θ)=∣A∣∣B∣A⋅B​
​可视化​：相似度热力图展示高关联酒店群

---

​三、行业应用案例​

案例1：西雅图酒店推荐系统

• ​数据特征​：名称、地址、描述（数据集下载）
• ​推荐逻辑​：

  def recommendations(name):
      idx = df[df['name']==name].index[0]
      sim_scores = list(enumerate(cosine_similarities[idx]))
      sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
      return df['name'].iloc[sim_scores[1:11]]

  ​效果​：

  输入酒店	推荐Top1	相似度
  Hilton Seattle Airport	Embassy Suites	0.87
  The Bacon Mansion	11th Avenue Inn	0.82

案例2：三国人物关系分析

# 加载预训练模型
model = Word2Vec.load('word2vec.model')
print(model.wv.most_similar('曹操', topn=5))
# 输出：[('曹丕', 0.92), ('夏侯惇', 0.88), ('荀彧', 0.85)...]

---

​四、技术选型指南​

1. Embedding模型对比

模型	维度	语言支持	适用场景
BGE-large-zh	1024	中文	电商客服
Jina-v4	2048	多语言	跨语言检索
Qwen3-8B	4096	中英	复杂语义理解

2. 向量数据库选型

---

​五、避坑指南​

1. ​维度灾难​：
   • 高维度（>1024）需配合PCA降维
2. ​冷启动问题​：
   • 混合使用TF-IDF和Embedding缓解稀疏问题
3. ​元数据管理​：

   # FAISS索引与元数据关联
   index = faiss.IndexIDMap(faiss.IndexFlatL2(1024))
   index.add_with_ids(vectors, ids)  # ids对应元数据表

---

​六、资源汇总​

• ​代码仓库​：包含酒店推荐和人物关系分析完整代码
• ​数据集​：西雅图酒店数据+三国演义文本
• ​模型下载​：预训练Word2Vec模型链接

---

案例实操

对三国演义的数据先进行 分词处理，使用jieba

!pip install -r requirements.txt 

gensim==4.3.3
jieba==0.42.1
scikit_learn==1.6.1

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: gensim==4.3.3 in /usr/local/lib/python3.11/site-packages (from -r requirements.txt (line 1)) (4.3.3)
Requirement already satisfied: jieba==0.42.1 in /usr/local/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (0.42.1)
Collecting scikit_learn==1.6.1 (from -r requirements.txt (line 3))
  Using cached scikit_learn-1.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Requirement already satisfied: numpy<2.0,>=1.18.5 in /usr/local/lib/python3.11/site-packages (from gensim==4.3.3->-r requirements.txt (line 1)) (1.26.4)
Requirement already satisfied: scipy<1.14.0,>=1.7.0 in /usr/local/lib/python3.11/site-packages (from gensim==4.3.3->-r requirements.txt (line 1)) (1.13.1)
Requirement already satisfied: smart-open>=1.8.1 in /usr/local/lib/python3.11/site-packages (from gensim==4.3.3->-r requirements.txt (line 1)) (7.3.0.post1)
Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/site-packages (from scikit_learn==1.6.1->-r requirements.txt (line 3)) (1.5.1)
Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/site-packages (from scikit_learn==1.6.1->-r requirements.txt (line 3)) (3.6.0)
Requirement already satisfied: wrapt in /usr/local/lib/python3.11/site-packages (from smart-open>=1.8.1->gensim==4.3.3->-r requirements.txt (line 1)) (1.17.2)
Downloading scikit_learn-1.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 179.5 kB/s eta 0:00:0000:0100:03
Installing collected packages: scikit_learn
Successfully installed scikit_learn-1.6.1

[notice] A new release of pip is available: 25.1.1 -> 25.2
[notice] To update, run: pip install --upgrade pip

 分词处理

import jieba
import os
import time
from utils import files_processing

source_folder = './three_kingdoms/source'
segment_folder = './three_kingdoms/segment'

def segment_lines(file_list, segment_out_dir, stopwords=None):
    if stopwords is None:
        stopwords = set()  # 默认空停用词表
    else:
        stopwords = set(stopwords)  # 转换为集合提高查询效率
    
    total_files = len(file_list)
    print(f"▶ 开始处理 {total_files} 个文件...")
    
    for i, file in enumerate(file_list):
        start_time = time.time()
        file_name = os.path.basename(file)
        segment_out_name = os.path.join(segment_out_dir, f'segment_{i}.txt')
        
        # 状态打印
        print(f"\n▌ 处理中 ({i+1}/{total_files}): {file_name}")
        
        try:
            # 优化：文本模式读取 + 指定UTF-8编码
            with open(file, 'r', encoding='utf-8') as f:
                document = f.read()
            
            # 精确模式分词 + 停用词过滤
            words = [
                word for word in jieba.cut(document) 
                if word.strip() and word not in stopwords
            ]
            result = ' '.join(words)
            
            # 写入分词结果
            with open(segment_out_name, 'w', encoding='utf-8') as f2:
                f2.write(result)
                
            # 耗时统计
            cost = time.time() - start_time
            print(f"✓ 完成! 生成词汇: {len(words)} 个, 耗时: {cost:.2f}秒")
            print(f"→ 输出路径: {segment_out_name}")
            
        except Exception as e:
            print(f"✗ 处理失败! 错误: {str(e)}")
    
    print(f"\n✅ 所有文件处理完成! 结果已保存至: {segment_out_dir}")

# 示例停用词表（需实际替换为你的停用词文件）
stopwords = ["的", "了", "和", "在", "是"]  # 示例数据

# 获取文件列表并执行
file_list = files_processing.get_files_list(source_folder, postfix='*.txt')
segment_lines(file_list, segment_folder, stopwords=stopwords)

 

处理前的数据 是整包【三国演义 作者：罗贯中】的文本

Jieba出来后的分词

 Word2Vec 训练

import jieba
import os
import multiprocessing
import shutil
from gensim.models import Word2Vec
from gensim.models.word2vec import PathLineSentences
from gensim.models.callbacks import CallbackAny2Vec

# 1. 清理干扰目录（解决IsADirectoryError核心问题）
def clean_temp_dirs(segment_folder):
    """删除.ipynb_checkpoints等临时目录"""
    checkpoint_dir = os.path.join(segment_folder, '.ipynb_checkpoints')
    if os.path.exists(checkpoint_dir):
        shutil.rmtree(checkpoint_dir, ignore_errors=True)

# 2. 增强版PathLineSentences（过滤隐藏文件和目录）
class FilteredPathLineSentences(PathLineSentences):
    def __init__(self, source):
        super().__init__(source)
        # 过滤隐藏文件和目录[2,3](@ref)
        self.input_files = [
            f for f in self.input_files 
            if not os.path.basename(f).startswith('.') 
            and os.path.isfile(f)
        ]

# 3. 训练进度监控
class EpochLogger(CallbackAny2Vec):
    def __init__(self):
        self.epoch = 0
    def on_epoch_begin(self, model):
        print(f"▶ 开始第 {self.epoch+1} 轮训练")
    def on_epoch_end(self, model):
        print(f"✓ 第 {self.epoch+1} 轮完成 | 当前词汇量: {len(model.wv)}")
        self.epoch += 1

# 主流程
def main():
    # 路径配置（使用绝对路径避免歧义）[3](@ref)
    segment_folder =  './three_kingdoms/segment'
    model_dir = './models'
    
    # 初始化环境
    os.makedirs(model_dir, exist_ok=True)
    clean_temp_dirs(segment_folder)

    # 语料加载（带过滤）
    try:
        sentences = FilteredPathLineSentences(segment_folder)
        print(f"✅ 成功加载 {len(sentences.input_files)} 个分词文件")
    except Exception as e:
        print(f"❌ 语料加载失败: {str(e)}")
        return

    # 模型训练（优化参数）[7,8](@ref)
    model = Word2Vec(
        sentences,
        vector_size=128,
        window=5,
        min_count=5,
        workers=multiprocessing.cpu_count(),
        epochs=10,
        sg=1,
        callbacks=[EpochLogger()]  # 训练进度监控
    )
    
    # 模型保存（双重备份）
    model.save('./models/word2Vec_3g.model')
    model.wv.save('./models/word_vectors_3g.kv')
    print(f"💾 模型已保存至 {model_dir}")

if __name__ == "__main__":
    main()

曹操的相似人物分析

 

# 加载词向量（无需加载完整模型）
from gensim.models import KeyedVectors
word_vectors = KeyedVectors.load('./models/word_vectors_3g.kv')

# 查找与曹操最相似的10个词（带相似度）
similar_to_caocao = word_vectors.most_similar('曹操', topn=10)
# === 语义分析 ===

# 1. 曹操相似词
print("曹操的相似人物:")
for word, score in word_vectors.most_similar('曹操', topn=10):
    print(f"{word}: {score:.4f}")

# 2. 向量运算分析
print("\n曹操+刘备-张飞 ≈ ")
results = word_vectors.most_similar(
    positive=['曹操', '刘备'],
    negative=['张飞'],
    topn=5
)
for word, score in results:
    print(f"{word}: {score:.4f}")    

曹操的相似人物:
操: 0.5050
孙权: 0.4963
今操: 0.4914
袁绍: 0.4775
马超: 0.4713
不及: 0.4455
孙策: 0.4432
陆逊: 0.4413
荀彧: 0.4395
吕布之: 0.4374

曹操+刘备-张飞 ≈ 
常有: 0.5848
荀彧: 0.5706
今操: 0.5663
不可不: 0.5541
西蜀: 0.5523