困惑度（Perplexity, PPL）

困惑度用于衡量语言模型对测试数据的预测能力，数值越低表示模型性能越好。计算公式基于交叉熵（Cross-Entropy）： [ \text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log P(w_i | w_{<i})\right) ] 其中：

• (N) 为测试集中的总词数；
• (P(w_i | w_{<i})) 是模型对第 (i) 个词的条件概率预测。

计算步骤：

1. 对测试集文本按词或子词单位分割；
2. 计算模型对每个词的条件概率对数；
3. 求均值后取指数得到困惑度。

BLEU（Bilingual Evaluation Understudy）

BLEU通过比较模型生成文本与参考文本的n-gram重叠度评估质量，常用1-gram到4-gram的加权几何平均： [ \text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{4} w_n \log p_n\right) ]

• (p_n) 为n-gram精度；
• (BP)（Brevity Penalty）惩罚过短输出： [ BP = \begin{cases} 1 & \text{if } c > r \ e^{1-r/c} & \text{if } c \leq r \end{cases} ] (c) 为生成文本长度，(r) 为参考文本的最接近长度。

计算步骤：

1. 统计生成文本与参考文本中1-gram到4-gram的匹配数量；
2. 计算各阶n-gram精度；
3. 应用短句惩罚后加权求和。

ROUGE（Recall-Oriented Understudy for Gisting Evaluation）

ROUGE侧重召回率，常用ROUGE-N（n-gram匹配）和ROUGE-L（最长公共子序列）：

ROUGE-N： [ \text{ROUGE-N} = \frac{\sum_{s \in S} \sum_{\text{n-gram} \in s} \text{Count}{\text{match}}(\text{n-gram})}{\sum{s \in S} \sum_{\text{n-gram} \in s} \text{Count}_{\text{reference}}(\text{n-gram})} ] 其中 (S) 为参考文本集。

ROUGE-L： 基于最长公共子序列（LCS）的F1值： [ R_{\text{LCS}} = \frac{LCS(X,Y)}{\text{len}(Y)}, \quad P_{\text{LCS}} = \frac{LCS(X,Y)}{\text{len}(X)}, \quad \text{ROUGE-L} = \frac{2R_{\text{LCS}}P_{\text{LCS}}}{R_{\text{LCS}}+P_{\text{LCS}}} ] (X) 为生成文本，(Y) 为参考文本。

计算步骤：

1. 对生成文本和参考文本提取n-gram或计算LCS；
2. 统计匹配数量并计算召回率、精确率；
3. 综合得分为F1值（ROUGE-L）或直接召回率（ROUGE-N）。

工具与库实现

• 困惑度：Hugging Face的transformers库可直接计算：

  from transformers import AutoModelForCausalLM, AutoTokenizer
  model = AutoModelForCausalLM.from_pretrained("gpt2")
  tokenizer = AutoTokenizer.from_pretrained("gpt2")
  inputs = tokenizer("sample text", return_tensors="pt")
  loss = model(**inputs, labels=inputs["input_ids"]).loss
  ppl = torch.exp(loss)
  

• BLEU/ROUGE：使用nltk或rouge库：

  from nltk.translate.bleu_score import sentence_bleu
  reference = [["this", "is", "a", "test"]]
  candidate = ["this", "is", "a", "prediction"]
  bleu = sentence_bleu(reference, candidate, weights=(0.25, 0.25, 0.25, 0.25))
  

• ROUGE：

  from rouge import Rouge
  rouge = Rouge()
  scores = rouge.get_scores("generated text", "reference text")
  

https://github.com/f6020/d/issues/1062

https://github.com/f6022/1/issues/1064

https://github.com/f6024/y/issues/1063

https://github.com/f6021/n/issues/1063

https://github.com/f6023/c/issues/1064

https://github.com/f6020/d/issues/1061

https://github.com/f6022/1/issues/1063

https://github.com/f6024/y/issues/1062

https://github.com/f6021/n/issues/1062

https://github.com/f6023/c/issues/1063

https://github.com/f6020/d/issues/1060

https://github.com/f6022/1/issues/1062

https://github.com/f6024/y/issues/1061

https://github.com/f6021/n/issues/1061

https://github.com/f6023/c/issues/1062

https://github.com/f6020/d/issues/1059

https://github.com/f6022/1/issues/1061

https://github.com/f6024/y/issues/1060

https://github.com/f6021/n/issues/1060

https://github.com/f6023/c/issues/1061

https://github.com/f6020/d/issues/1058

https://github.com/f6022/1/issues/1060

https://github.com/f6024/y/issues/1059

https://github.com/f6021/n/issues/1059

https://github.com/f6023/c/issues/1060

https://github.com/f6020/d/issues/1057

https://github.com/f6022/1/issues/1059

https://github.com/f6024/y/issues/1058

https://github.com/f6021/n/issues/1058

https://github.com/f6023/c/issues/1059

https://github.com/f6020/d/issues/1056

https://github.com/f6022/1/issues/1058

https://github.com/f6024/y/issues/1057

https://github.com/f6021/n/issues/1057

https://github.com/f6023/c/issues/1058

https://github.com/f6020/d/issues/1055

https://github.com/f6022/1/issues/1057

https://github.com/f6024/y/issues/1056

https://github.com/f6021/n/issues/1056

https://github.com/f6023/c/issues/1057

https://github.com/f6020/d/issues/1054

https://github.com/f6022/1/issues/1056

https://github.com/f6024/y/issues/1055

https://github.com/f6021/n/issues/1055

https://github.com/f6023/c/issues/1056

https://github.com/f6020/d/issues/1053

https://github.com/f6022/1/issues/1055

https://github.com/f6024/y/issues/1054

https://github.com/f6021/n/issues/1054

https://github.com/f6023/c/issues/1055

https://github.com/f6020/d/issues/1052

https://github.com/f6022/1/issues/1054

https://github.com/f6024/y/issues/1053

https://github.com/f6021/n/issues/1053

https://github.com/f6023/c/issues/1054

https://github.com/f6020/d/issues/1051

https://github.com/f6022/1/issues/1053

https://github.com/f6024/y/issues/1052

https://github.com/f6021/n/issues/1052

https://github.com/f6023/c/issues/1053

https://github.com/f6020/d/issues/1050

https://github.com/f6022/1/issues/1052

https://github.com/f6024/y/issues/1051

https://github.com/f6021/n/issues/1051

https://github.com/f6023/c/issues/1052

https://github.com/f6020/d/issues/1049

https://github.com/f6022/1/issues/1051

https://github.com/f6024/y/issues/1050

https://github.com/f6021/n/issues/1050

https://github.com/f6023/c/issues/1051

https://github.com/f6020/d/issues/1048

https://github.com/f6022/1/issues/1050

https://github.com/f6024/y/issues/1049

https://github.com/f6021/n/issues/1049

https://github.com/f6023/c/issues/1050

https://github.com/f6020/d/issues/1047

https://github.com/f6022/1/issues/1049

https://github.com/f6024/y/issues/1048

https://github.com/f6021/n/issues/1048

https://github.com/f6023/c/issues/1049

https://github.com/f6020/d/issues/1046

https://github.com/f6022/1/issues/1048

https://github.com/f6024/y/issues/1047

https://github.com/f6021/n/issues/1047

https://github.com/f6023/c/issues/1048

https://github.com/f6020/d/issues/1045

https://github.com/f6022/1/issues/1047

https://github.com/f6024/y/issues/1046

https://github.com/f6021/n/issues/1046

https://github.com/f6023/c/issues/1047

https://github.com/f6020/d/issues/1044

https://github.com/f6022/1/issues/1046

https://github.com/f6024/y/issues/1045

https://github.com/f6021/n/issues/1045

https://github.com/f6023/c/issues/1046

https://github.com/f6020/d/issues/1043

https://github.com/f6022/1/issues/1045

https://github.com/f6024/y/issues/1044

https://github.com/f6021/n/issues/1044

https://github.com/f6023/c/issues/1045

https://github.com/f6020/d/issues/1042

https://github.com/f6022/1/issues/1044

https://github.com/f6024/y/issues/1043

https://github.com/f6021/n/issues/1043

https://github.com/f6023/c/issues/1044

https://github.com/f6020/d/issues/1041

https://github.com/f6022/1/issues/1043

https://github.com/f6021/n/issues/1042

https://github.com/f6024/y/issues/1042

https://github.com/f6023/c/issues/1043

https://github.com/f6020/d/issues/1040

https://github.com/f6022/1/issues/1042

https://github.com/f6021/n/issues/1041

https://github.com/f6024/y/issues/1041

https://github.com/f6023/c/issues/1042

https://github.com/f6020/d/issues/1039

https://github.com/f6022/1/issues/1041

https://github.com/f6021/n/issues/1040

https://github.com/f6024/y/issues/1040

https://github.com/f6023/c/issues/1041

https://github.com/f6020/d/issues/1038

https://github.com/f6022/1/issues/1040

https://github.com/f6021/n/issues/1039

https://github.com/f6024/y/issues/1039

https://github.com/f6023/c/issues/1040

https://github.com/f6020/d/issues/1037

https://github.com/f6022/1/issues/1039

https://github.com/f6021/n/issues/1038

https://github.com/f6024/y/issues/1038

https://github.com/f6023/c/issues/1039

https://github.com/f6020/d/issues/1036

https://github.com/f6022/1/issues/1038

https://github.com/f6021/n/issues/1037

https://github.com/f6024/y/issues/1037

https://github.com/f6023/c/issues/1038

https://github.com/f6020/d/issues/1035

https://github.com/f6022/1/issues/1037

https://github.com/f6021/n/issues/1036

https://github.com/f6024/y/issues/1036

https://github.com/f6023/c/issues/1037

https://github.com/f6020/d/issues/1034

https://github.com/f6022/1/issues/1036

https://github.com/f6021/n/issues/1035

https://github.com/f6024/y/issues/1035

https://github.com/f6023/c/issues/1036

https://github.com/f6020/d/issues/1033

https://github.com/f6022/1/issues/1035

https://github.com/f6021/n/issues/1034

https://github.com/f6024/y/issues/1034

https://github.com/f6023/c/issues/1035

https://github.com/f6020/d/issues/1032

https://github.com/f6022/1/issues/1034

https://github.com/f6021/n/issues/1033

https://github.com/f6024/y/issues/1033

https://github.com/f6023/c/issues/1034

https://github.com/f6020/d/issues/1031

https://github.com/f6022/1/issues/1033

https://github.com/f6021/n/issues/1032

https://github.com/f6024/y/issues/1032

https://github.com/f6023/c/issues/1033

https://github.com/f6020/d/issues/1030

https://github.com/f6022/1/issues/1032

https://github.com/f6021/n/issues/1031

https://github.com/f6024/y/issues/1031

https://github.com/f6023/c/issues/1032

https://github.com/f6020/d/issues/1029

https://github.com/f6022/1/issues/1031

https://github.com/f6021/n/issues/1030

https://github.com/f6024/y/issues/1030

https://github.com/f6023/c/issues/1031

https://github.com/f6020/d/issues/1028

https://github.com/f6022/1/issues/1030

https://github.com/f6021/n/issues/1029

https://github.com/f6024/y/issues/1029

https://github.com/f6023/c/issues/1030

https://github.com/f6020/d/issues/1027

https://github.com/f6022/1/issues/1029

https://github.com/f6021/n/issues/1028

https://github.com/f6024/y/issues/1028

https://github.com/f6023/c/issues/1029

https://github.com/f6020/d/issues/1026

https://github.com/f6022/1/issues/1028

https://github.com/f6021/n/issues/1027

https://github.com/f6024/y/issues/1027

https://github.com/f6023/c/issues/1028

https://github.com/f6020/d/issues/1025

https://github.com/f6022/1/issues/1027

https://github.com/f6021/n/issues/1026

https://github.com/f6024/y/issues/1026

https://github.com/f6023/c/issues/1027

https://github.com/f6020/d/issues/1024

https://github.com/f6022/1/issues/1026

https://github.com/f6021/n/issues/1025

https://github.com/f6024/y/issues/1025

https://github.com/f6023/c/issues/1026

https://github.com/f6020/d/issues/1023

https://github.com/f6022/1/issues/1025

https://github.com/f6021/n/issues/1024

https://github.com/f6024/y/issues/1024

https://github.com/f6023/c/issues/1025

https://github.com/f6020/d/issues/1022

https://github.com/f6022/1/issues/1024

https://github.com/f6021/n/issues/1023

https://github.com/f6024/y/issues/1023

https://github.com/f6023/c/issues/1024

https://github.com/f6020/d/issues/1021

https://github.com/f6022/1/issues/1023

https://github.com/f6021/n/issues/1022

https://github.com/f6024/y/issues/1022

https://github.com/f6023/c/issues/1023

https://github.com/f6020/d/issues/1020

https://github.com/f6022/1/issues/1022

https://github.com/f6021/n/issues/1021

https://github.com/f6024/y/issues/1021

https://github.com/f6023/c/issues/1022

https://github.com/f6020/d/issues/1019

https://github.com/f6022/1/issues/1021

https://github.com/f6021/n/issues/1020

https://github.com/f6024/y/issues/1020

https://github.com/f6023/c/issues/1021

https://github.com/f6020/d/issues/1018

https://github.com/f6022/1/issues/1020

https://github.com/f6021/n/issues/1019

https://github.com/f6024/y/issues/1019

https://github.com/f6023/c/issues/1020

https://github.com/f6020/d/issues/1017

https://github.com/f6022/1/issues/1019

https://github.com/f6021/n/issues/1018

https://github.com/f6024/y/issues/1018

https://github.com/f6023/c/issues/1019

https://github.com/f6020/d/issues/1016

https://github.com/f6022/1/issues/1018

https://github.com/f6021/n/issues/1017

https://github.com/f6024/y/issues/1017

https://github.com/f6023/c/issues/1018

https://github.com/f6020/d/issues/1015

https://github.com/f6022/1/issues/1017

https://github.com/f6021/n/issues/1016

https://github.com/f6024/y/issues/1016

https://github.com/f6023/c/issues/1017

https://github.com/f6020/d/issues/1014

https://github.com/f6022/1/issues/1016

https://github.com/f6021/n/issues/1015

https://github.com/f6024/y/issues/1015

https://github.com/f6023/c/issues/1016

https://github.com/f6020/d/issues/1013

https://github.com/f6022/1/issues/1015

https://github.com/f6021/n/issues/1014

https://github.com/f6024/y/issues/1014

https://github.com/f6023/c/issues/1015

https://github.com/f6020/d/issues/1012

https://github.com/f6022/1/issues/1014

https://github.com/f6021/n/issues/1013

https://github.com/f6024/y/issues/1013

https://github.com/f6023/c/issues/1014

https://github.com/f6020/d/issues/1011

https://github.com/f6022/1/issues/1013

https://github.com/f6021/n/issues/1012

https://github.com/f6024/y/issues/1012

https://github.com/f6023/c/issues/1013

https://github.com/f6020/d/issues/1010

https://github.com/f6022/1/issues/1012

https://github.com/f6021/n/issues/1011

https://github.com/f6024/y/issues/1011

https://github.com/f6023/c/issues/1012

https://github.com/f6020/d/issues/1009

https://github.com/f6022/1/issues/1011

https://github.com/f6021/n/issues/1010

https://github.com/f6024/y/issues/1010

https://github.com/f6023/c/issues/1011

https://github.com/f6020/d/issues/1008

https://github.com/f6022/1/issues/1010

https://github.com/f6021/n/issues/1009

https://github.com/f6024/y/issues/1009

https://github.com/f6023/c/issues/1010

https://github.com/f6020/d/issues/1007

https://github.com/f6022/1/issues/1009

https://github.com/f6021/n/issues/1008

https://github.com/f6024/y/issues/1008

https://github.com/f6023/c/issues/1009

https://github.com/f6020/d/issues/1006

https://github.com/f6022/1/issues/1008

https://github.com/f6021/n/issues/1007

https://github.com/f6024/y/issues/1007

https://github.com/f6023/c/issues/1008

https://github.com/f6020/d/issues/1005

https://github.com/f6022/1/issues/1007

https://github.com/f6024/y/issues/1006

https://github.com/f6021/n/issues/1006

https://github.com/f6023/c/issues/1007

https://github.com/f6020/d/issues/1004

https://github.com/f6022/1/issues/1006

https://github.com/f6024/y/issues/1005

https://github.com/f6021/n/issues/1005

https://github.com/f6023/c/issues/1006

https://github.com/f6020/d/issues/1003

https://github.com/f6022/1/issues/1005

https://github.com/f6024/y/issues/1004

https://github.com/f6021/n/issues/1004

https://github.com/f6023/c/issues/1005

https://github.com/f6020/d/issues/1002

https://github.com/f6022/1/issues/1004

https://github.com/f6024/y/issues/1003

https://github.com/f6021/n/issues/1003

https://github.com/f6023/c/issues/1004

https://github.com/f6020/d/issues/1001

https://github.com/f6022/1/issues/1003

https://github.com/f6021/n/issues/1002

https://github.com/f6024/y/issues/1002

https://github.com/f6023/c/issues/1003

https://github.com/f6020/d/issues/1000

https://github.com/f6022/1/issues/1002

https://github.com/f6021/n/issues/1001

https://github.com/f6024/y/issues/1001

https://github.com/f6023/c/issues/1002

https://github.com/f6020/d/issues/999

https://github.com/f6022/1/issues/1001

https://github.com/f6021/n/issues/1000

https://github.com/f6024/y/issues/1000

https://github.com/f6023/c/issues/1001

https://github.com/f6020/d/issues/998

https://github.com/f6022/1/issues/1000

https://github.com/f6021/n/issues/999

https://github.com/f6024/y/issues/999

https://github.com/f6023/c/issues/1000

https://github.com/f6020/d/issues/997

https://github.com/f6022/1/issues/999

https://github.com/f6021/n/issues/998

https://github.com/f6024/y/issues/998

https://github.com/f6023/c/issues/999

https://github.com/f6020/d/issues/996

https://github.com/f6022/1/issues/998

https://github.com/f6021/n/issues/997

https://github.com/f6024/y/issues/997

https://github.com/f6023/c/issues/998

https://github.com/f6020/d/issues/995

https://github.com/f6022/1/issues/997

https://github.com/f6021/n/issues/996

https://github.com/f6024/y/issues/996

https://github.com/f6023/c/issues/997

https://github.com/f6020/d/issues/994

https://github.com/f6022/1/issues/996

https://github.com/f6021/n/issues/995

https://github.com/f6024/y/issues/995

https://github.com/f6023/c/issues/996

https://github.com/f6020/d/issues/993

https://github.com/f6022/1/issues/995

https://github.com/f6021/n/issues/994

https://github.com/f6024/y/issues/994

https://github.com/f6023/c/issues/995

https://github.com/f6020/d/issues/992

https://github.com/f6022/1/issues/994

https://github.com/f6021/n/issues/993

https://github.com/f6024/y/issues/993

https://github.com/f6023/c/issues/994

https://github.com/f6020/d/issues/991

https://github.com/f6022/1/issues/993

https://github.com/f6021/n/issues/992

https://github.com/f6024/y/issues/992

https://github.com/f6023/c/issues/993

https://github.com/f6020/d/issues/990

https://github.com/f6022/1/issues/992

https://github.com/f6021/n/issues/991

https://github.com/f6024/y/issues/991

https://github.com/f6023/c/issues/992

https://github.com/f6020/d/issues/989

https://github.com/f6022/1/issues/991

https://github.com/f6021/n/issues/990

https://github.com/f6024/y/issues/990

https://github.com/f6023/c/issues/991

https://github.com/f6020/d/issues/988

https://github.com/f6022/1/issues/990

https://github.com/f6021/n/issues/989

https://github.com/f6024/y/issues/989

https://github.com/f6023/c/issues/990

https://github.com/f6020/d/issues/987

https://github.com/f6022/1/issues/989

https://github.com/f6021/n/issues/988

https://github.com/f6024/y/issues/988

https://github.com/f6023/c/issues/989

https://github.com/f6020/d/issues/986

https://github.com/f6022/1/issues/988

https://github.com/f6021/n/issues/987

https://github.com/f6024/y/issues/987

https://github.com/f6023/c/issues/988

https://github.com/f6020/d/issues/985

https://github.com/f6022/1/issues/987

https://github.com/f6021/n/issues/986

https://github.com/f6024/y/issues/986

https://github.com/f6023/c/issues/987

https://github.com/f6020/d/issues/984

https://github.com/f6022/1/issues/986

https://github.com/f6021/n/issues/985

https://github.com/f6024/y/issues/985

https://github.com/f6023/c/issues/986

https://github.com/f6020/d/issues/983

https://github.com/f6022/1/issues/985

https://github.com/f6021/n/issues/984

https://github.com/f6024/y/issues/984

https://github.com/f6023/c/issues/985

https://github.com/f6020/d/issues/982

https://github.com/f6022/1/issues/984

https://github.com/f6021/n/issues/983

https://github.com/f6024/y/issues/983

https://github.com/f6023/c/issues/984

https://github.com/f6020/d/issues/981

https://github.com/f6022/1/issues/983

https://github.com/f6021/n/issues/982

https://github.com/f6024/y/issues/982

https://github.com/f6023/c/issues/983

https://github.com/f6020/d/issues/980

https://github.com/f6022/1/issues/982

https://github.com/f6021/n/issues/981

https://github.com/f6024/y/issues/981

https://github.com/f6023/c/issues/982

https://github.com/f6020/d/issues/979

https://github.com/f6022/1/issues/981

https://github.com/f6021/n/issues/980

https://github.com/f6024/y/issues/980

https://github.com/f6023/c/issues/981

https://github.com/f6020/d/issues/978

https://github.com/f6022/1/issues/980

https://github.com/f6021/n/issues/979

https://github.com/f6024/y/issues/979

https://github.com/f6023/c/issues/980

https://github.com/f6020/d/issues/977

https://github.com/f6022/1/issues/979

https://github.com/f6021/n/issues/978

https://github.com/f6024/y/issues/978

https://github.com/f6023/c/issues/979

https://github.com/f6020/d/issues/976

https://github.com/f6022/1/issues/978

https://github.com/f6021/n/issues/977

https://github.com/f6024/y/issues/977

https://github.com/f6023/c/issues/978

https://github.com/f6020/d/issues/975

https://github.com/f6022/1/issues/977

https://github.com/f6021/n/issues/976

https://github.com/f6024/y/issues/976

https://github.com/f6023/c/issues/977

https://github.com/f6020/d/issues/974

https://github.com/f6022/1/issues/976

https://github.com/f6021/n/issues/975

https://github.com/f6024/y/issues/975

https://github.com/f6023/c/issues/976

https://github.com/f6020/d/issues/973

https://github.com/f6022/1/issues/975

https://github.com/f6021/n/issues/974

https://github.com/f6024/y/issues/974

https://github.com/f6023/c/issues/975

https://github.com/f6020/d/issues/972

https://github.com/f6022/1/issues/974

https://github.com/f6021/n/issues/973

https://github.com/f6024/y/issues/973

https://github.com/f6023/c/issues/974

https://github.com/f6020/d/issues/971

https://github.com/f6022/1/issues/973

https://github.com/f6021/n/issues/972

https://github.com/f6024/y/issues/972

https://github.com/f6023/c/issues/973

https://github.com/f6020/d/issues/970

https://github.com/f6022/1/issues/972

https://github.com/f6021/n/issues/971

https://github.com/f6024/y/issues/971

https://github.com/f6023/c/issues/972

https://github.com/f6020/d/issues/969

https://github.com/f6022/1/issues/971

https://github.com/f6021/n/issues/970

https://github.com/f6024/y/issues/970

https://github.com/f6023/c/issues/971

https://github.com/f6020/d/issues/968

https://github.com/f6022/1/issues/970

https://github.com/f6021/n/issues/969

https://github.com/f6024/y/issues/969

https://github.com/f6023/c/issues/970

https://github.com/f6020/d/issues/967

https://github.com/f6022/1/issues/969

https://github.com/f6021/n/issues/968

https://github.com/f6024/y/issues/968

https://github.com/f6023/c/issues/969

https://github.com/f6020/d/issues/966

https://github.com/f6022/1/issues/968

https://github.com/f6021/n/issues/967

https://github.com/f6024/y/issues/967

https://github.com/f6023/c/issues/968

https://github.com/f6020/d/issues/965

https://github.com/f6022/1/issues/967

https://github.com/f6021/n/issues/966

https://github.com/f6024/y/issues/966

https://github.com/f6023/c/issues/967

https://github.com/f6020/d/issues/964

https://github.com/f6022/1/issues/966

https://github.com/f6021/n/issues/965

https://github.com/f6024/y/issues/965

https://github.com/f6023/c/issues/966

https://github.com/f6020/d/issues/963

https://github.com/f6022/1/issues/965

https://github.com/f6021/n/issues/964

https://github.com/f6024/y/issues/964

https://github.com/f6023/c/issues/965

https://github.com/f6020/d/issues/962

https://github.com/f6022/1/issues/964

https://github.com/f6021/n/issues/963

https://github.com/f6024/y/issues/963

https://github.com/f6023/c/issues/964

https://github.com/f6020/d/issues/961

https://github.com/f6022/1/issues/963

https://github.com/f6021/n/issues/962

https://github.com/f6024/y/issues/962

https://github.com/f6023/c/issues/963

https://github.com/f6020/d/issues/960

https://github.com/f6022/1/issues/962

https://github.com/f6021/n/issues/961

https://github.com/f6024/y/issues/961

https://github.com/f6023/c/issues/962

https://github.com/f6020/d/issues/959

https://github.com/f6022/1/issues/961

https://github.com/f6021/n/issues/960

https://github.com/f6024/y/issues/960

https://github.com/f6023/c/issues/961

https://github.com/f6020/d/issues/958

https://github.com/f6022/1/issues/960

https://github.com/f6021/n/issues/959

https://github.com/f6024/y/issues/959

https://github.com/f6023/c/issues/960

https://github.com/f6020/d/issues/957

https://github.com/f6022/1/issues/959

https://github.com/f6021/n/issues/958

https://github.com/f6024/y/issues/958

https://github.com/f6023/c/issues/959

https://github.com/f6020/d/issues/956

https://github.com/f6022/1/issues/958

https://github.com/f6021/n/issues/957

https://github.com/f6024/y/issues/957

https://github.com/f6023/c/issues/958

https://github.com/f6020/d/issues/955

https://github.com/f6022/1/issues/957

https://github.com/f6021/n/issues/956

https://github.com/f6024/y/issues/956

https://github.com/f6023/c/issues/957

https://github.com/f6020/d/issues/954

https://github.com/f6022/1/issues/956

https://github.com/f6021/n/issues/955

https://github.com/f6024/y/issues/955

https://github.com/f6023/c/issues/956

https://github.com/f6020/d/issues/953

https://github.com/f6022/1/issues/955

https://github.com/f6021/n/issues/954

https://github.com/f6024/y/issues/954

https://github.com/f6023/c/issues/955

https://github.com/f6020/d/issues/952

https://github.com/f6022/1/issues/954

https://github.com/f6021/n/issues/953

https://github.com/f6024/y/issues/953

https://github.com/f6023/c/issues/954

https://github.com/f6020/d/issues/951

https://github.com/f6022/1/issues/953

https://github.com/f6021/n/issues/952

https://github.com/f6024/y/issues/952

https://github.com/f6023/c/issues/953

https://github.com/f6020/d/issues/950

https://github.com/f6022/1/issues/952

https://github.com/f6021/n/issues/951

https://github.com/f6024/y/issues/951

https://github.com/f6023/c/issues/952

https://github.com/f6020/d/issues/949

https://github.com/f6022/1/issues/951

https://github.com/f6021/n/issues/950

https://github.com/f6024/y/issues/950

https://github.com/f6023/c/issues/951

https://github.com/f6020/d/issues/948

https://github.com/f6022/1/issues/950