1 系统环境

硬件环境(Ascend/GPU/CPU): Ascend

MindSpore版本: 2.1

执行模式（PyNative/ Graph）: 不限

2 报错信息

2.1 问题描述

加载WizardCoderTokenizer.from_pretrained(…)时出现报错，发现缺少两个文件vocab_file和merge_file。

Traceback (most recent call last):   File /home/wizardcoder/1_wizardcoder-mindformers/mindformers/tools/register/register.py" , line 217, in get_instance return obi cls(**kwaras)
TypeError:__init_() missing 2 required positional arguments: 'vocab_file' and 'merge_file'复制

3 根因分析

在tokenizer_config.json文件新增两列，指定这两个文件路径

4 解决方案

需在tokenizer_config.json文件中增加vocab_file和merge_file对应的文件路径。

"bos_token": "<lendoftext|>",
"clean_up_tokenization_spaces": true,
"eos token": "＜lendoftext|>",
"model_max_length": 2048,
"padding_side": "right",
"tokenizer_class": "WizardCoderTokenizer",
"unk_token": "<lendoftext|>",
"vocab size": 49152,
"vocab_file": "/xxxxxx/wizardcoder/vocab.json",
"merge_file":  '/xxxxxx/wizardcoder/merges.txt