通常通过镜像网站下载https://hf-mirror.com/
在链接页面有介绍方法,对于不大的模型可以直接下载。这里介绍比较常用且方便的下载方法。

使用huggingface 官方提供的 huggingface-cli 命令行工具

安装(huggingface_hub、hf_transfer安装可以使用-i命令从镜像网站下载)

pip install -U huggingface_hub hf_transfer

基本命令(每次打开远程链接都要输入)

export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_ENDPOINT=https://hf-mirror.com
# 设置本地cache地址,可忽略
export HUGGINGFACE_HUB_CACHE=/data/models/cache

下载模型(下载NousResearch/Llama-2-13b-chat-hf,放在当前目录文件夹Llama-2-13b-chat-hf下)

huggingface-cli download --resume-download NousResearch/Llama-2-13b-chat-hf --local-dir Llama-2-13b-chat-hf --local-dir-use-symlinks False
HF_HUB_ENABLE_HF_TRANSFER=0 huggingface-cli download --resume-download tsystems/colqwen2.5-3b-multilingual-v1.0 --local-dir tsystems/colqwen2.5-3b-multilingual-v1.0 --local-dir-use-symlinks False
HF_HUB_ENABLE_HF_TRANSFER=0 hf download tsystems/colqwen2.5-3b-multilingual-v1.0 --local-dir tsystems/colqwen2.5-3b-multilingual-v1.0 --local-dir-use-symlinks False

下载数据集(需补充参数 --repo-type dataset)

HF_HUB_ENABLE_HF_TRANSFER=0 huggingface-cli download --resume-download Episoode/Double-Bench --local-dir Double-Bench --local-dir-use-symlinks False --repo-type dataset

携带参数下载

huggingface-cli download --token hf_*** --resume-download meta-llama/Llama-2-7b-hf --local-dir Llama-2-7b-hf --local-dir-use-symlinks False

对于下载中断的问题,由于huggingface-cli本身能断点重传,但下载界面关闭后只能重新下载,因而采用选择性下载

huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "model-00013-of-00015.safetensors"

huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "pytorch_model*.bin"

可直接下载到cache中(export HUGGINGFACE_HUB_CACHE=/data/models/cache这里所指定的)

huggingface-cli download NousResearch/Llama-2-70b-chat-hf

其他用法

huggingface-cli download --helps

usage: huggingface-cli <command> [<args>] download [-h] [--repo-type {model,dataset,space}]
                                                   [--revision REVISION] [--include [INCLUDE ...]]
                                                   [--exclude [EXCLUDE ...]] [--cache-dir CACHE_DIR]  
                                                   [--local-dir LOCAL_DIR]   
                                                   [--local-dir-use-symlinks {auto,True,False}]   
                                                   [--force-download] [--resume-download]
                                                   [--token TOKEN]                                                         
                                                   [--quiet]                                                                                                      
                                                   repo_id [filenames ...]
positional arguments:                                                                                            repo_id               ID of the repo to download from (e.g. `username/repo-name`).
  filenames             Files to download (e.g. `config.json`, `data/metadata.jsonl`).

options:
  -h, --help            show this help message and exit
  --repo-type {model,dataset,space}
                        Type of repo to download from (e.g. `dataset`).
  --revision REVISION   An optional Git revision id which can be a branch name, a tag, or a commit hash.
  --include [INCLUDE ...]
                        Glob patterns to match files to download.
  --exclude [EXCLUDE ...]
                        Glob patterns to exclude from files to download.
  --cache-dir CACHE_DIR
                        Path to the directory where to save the downloaded files.
  --local-dir LOCAL_DIR
                        If set, the downloaded file will be placed under this directory either as a symlink
                        (default) or a regular file. Check out
                        https://huggingface.co/docs/huggingface_hub/guides/download#download-files-to-local-
                        folder for more details.
  --local-dir-use-symlinks {auto,True,False}
                        To be used with `local_dir`. If set to 'auto', the cache directory will be used and
                        the file will be either duplicated or symlinked to the local directory depending on
                        its size. It set to `True`, a symlink will be created, no matter the file size. If
                        set to `False`, the file will either be duplicated from cache (if already exists) or
                        downloaded from the Hub and not cached.
  --force-download      If True, the files will be downloaded even if they are already cached.
  --resume-download     If True, resume a previously interrupted download.
  --token TOKEN         A User Access Token generated from https://huggingface.co/settings/tokens
  --quiet               If True, progress bars are disabled and only the path to the download files is
                        printed.

使用hfd下载

下载hfd并增加权限

wget https://hf-mirror.com/hfd/hfd.sh
chmod a+x hfd.sh

基本命令(每次打开远程链接都要输入)

export HF_ENDPOINT=https://hf-mirror.com

下载模型

./hfd.sh Qwen2.5-VL-3B-Instruct

下载数据集

./hfd.sh wikitext --dataset

魔搭社区

国内可用,已经同步了大部分热门模型
安装

export MODELSCOPE_CACHE=/home/npy/nas
pip install modelscope

下载模型(注意下载名称与原来可能不同)

modelscope download --model=Qwen/Qwen2.5-VL-3B-Instruct --local_dir Qwen/Qwen2.5-VL-3B-Instruct

下载数据集(注意下载名称与原来可能不同)

modelscope download --dataset=modelscope/CLUE --local_dir ./CLUE
Logo

有“AI”的1024 = 2048,欢迎大家加入2048 AI社区

更多推荐