leftyer/mindformers
1
0
Fork 0
mirror of https://gitee.com/mindspore/mindformers.git synced 2025-12-06 11:29:59 +08:00
Code Issues Packages Projects Releases Wiki Activity

add url statement

Browse Source
This commit is contained in:
zyw_hw
2025-09-23 10:48:53 +08:00
parent 0e7c6baf65
commit 7d163f32f2
6 changed files with 41 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.jenkins/check/config/whitelizard.txt
View File

@@ -42,4 +42,5 @@ mindformers/mindformers/tools/hub/dynamic_module_utils.py:get_cached_module_file
mindformers/mindformers/tools/hub/dynamic_module_utils.py:get_class_from_dynamic_module
mindformers/mindformers/models/llama/llama_config.py:__init__
mindformers/research/deepseek3/deepseek3_config.py:__init__
mindformers/research/telechat2/telechat_config.py:__init__
mindformers/research/telechat2/telechat_config.py:__init__
mindformers/mindformers/tools/hub/hub.py:cached_file

31
docs/security_statement.md
View File

@@ -50,19 +50,24 @@ mindformers在源码构建安装过程中，会产生临时目录和程序文件
### 公网地址声明
| 类型 | 开源代码地址 | 文件名 | 公网IP地址/公网URL地址/域名/邮箱地址 | 用途说明 |
|:----:|:------:|:-----------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------:|
| 开发引入 | - | mindformers/build.sh | https://pypi.tuna.tsinghua.edu.cn/simple | 用于pip安装mindformers时, 修改安装源 |
| 开发引入 | - | mindformers/.jenkins/test/config/dependent_packages.yaml | https://repo.mindspore.cn/mindspore/mindspore/version/202502/20250214/master_20250214091349_bba9269b8a6826c23ec33be9fd677178d52cfa3f_newest/ | 用于下载测试所依赖的mindspore包 |
| 开发引入 | - | mindformers/mindformers/tools/hub/hub.py | https://hub-ci.openmind.cn | staging_mode下用于从openmind下载模型 |
| 开发引入 | - | mindformers/mindformers/tools/hub/hub.py | https://openmind.cn | 非staging_mode下用于从openmind下载模型 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers | setup脚本方式安装mindformers地址 |
| 开发引入 | - | mindformers/trainer/training_args.py | https://www.mindspore.cn/mindformers/docs/zh-CN/master/feature/ckpt.html#%E6%9D%83%E9%87%8D%E5%88%87%E5%88%86%E4%B8%8E%E5%90%88%E5%B9%B6 | mindformers权重切分与合并特性文档 |
| 开发引入 | - | mindformers/run_check.py | https://www.hiascend.com/document/detail/zh/canncommercial/ | CANN官方使用文档 |
| 开发引入 | - | mindformers/run_check.py | https://www.mindspore.cn/install/ | mindspore官方安装文档 |
| 开发引入 | - | run_mindformer.py | https://gitee.com/mindspore/mindformers#%E4%BB%8B%E7%BB%8D | mindformers介绍 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers/tags | mindformers包下载链接 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers/issues | mindformers的issue链接 |
| 类型 | 开源代码地址 | 文件名 | 公网IP地址/公网URL地址/域名/邮箱地址 | 用途说明 |
|:----:|:------:|:--------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------:|
| 开发引入 | - | mindformers/.jenkins/test/config/dependent_packages.yaml | https://repo.mindspore.cn/mindspore/mindspore/version/202508/20250830/master_20250830161002_e6ff5e16e936d4238dc1b444b288fe55c39ac2e4_newest/ | 用于下载测试所依赖的mindspore包 |
| 开发引入 | - | mindformers/build.sh | https://pypi.tuna.tsinghua.edu.cn/simple | 用于pip安装mindformers时, 修改安装源 |
| 开发引入 | - | mindformers/core/callback/callback.py | https://www.hiascend.com/document/detail/zh/mindstudio/80RC1/ | MindStudio官网文档链接 |
| 开发引入 | - | mindformers/models/processing_utils.py | https://modelers.cn/models | 魔乐社区模型库链接 |
| 开发引入 | - | mindformers/models/tokenization_utils_base.py | https://modelers.cn/models | 魔乐社区模型库链接 |
| 开发引入 | - | mindformers/run_check.py | https://gitee.com/mindspore/mindformers/issues | mindformers的issue链接 |
| 开发引入 | - | mindformers/run_check.py | https://www.hiascend.com/document/detail/zh/canncommercial/ | CANN官方使用文档 |
| 开发引入 | - | mindformers/run_check.py | https://www.mindspore.cn/install/ | mindspore官方安装文档 |
| 开发引入 | - | mindformers/mindformers/tools/hub/hub.py | https://modelers.cn/ | 魔乐社区官网链接 |
| 开发引入 | - | mindformers/trainer/training_args.py | https://www.mindspore.cn/mindformers/docs/zh-CN/master/feature/ckpt.html#%E6%9D%83%E9%87%8D%E5%88%87%E5%88%86%E4%B8%8E%E5%90%88%E5%B9%B6 | mindformers权重切分与合并特性文档 |
| 开发引入 | - | run_mindformer.py | https://gitee.com/mindspore/mindformers#%E4%BB%8B%E7%BB%8D | mindformers介绍 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers/tags | mindformers包下载链接 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers/issues | mindformers的issue链接 |
| 开发引入 | - | mindformers/setup.py | https://www.mindspore.cn | mindspore官网链接 |
| 开发引入 | - | mindformers/setup.py | contact@mindspore.cn | mindspore的联系邮箱 |
| 开发引入 | - | mindformers/setup.py | https://gitee.com/mindspore/mindformers | setup脚本方式安装mindformers地址 |
### 公开接口声明

4
mindformers/models/processing_utils.py
View File

@@ -280,7 +280,7 @@ class ProcessorMixin(PushToHubMixin):
pretrained_model_name_or_path (`str` or `os.PathLike`):
The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
subfolder (`str`, *optional*, defaults to `""`):
In case the relevant files are located inside a subfolder of the model repo on openmind.cn, you can
In case the relevant files are located inside a subfolder of the model repo on modelers.cn, you can
specify the folder name here.
Returns:
@@ -339,7 +339,7 @@ class ProcessorMixin(PushToHubMixin):
# For any other exception, we throw a generic error.
raise EnvironmentError(
f"Can't load processor for '{pretrained_model_name_or_path}'. If you were trying to load"
" it from 'https://openmind.cn/models', make sure you don't have a local directory with the"
" it from 'https://modelers.cn/models', make sure you don't have a local directory with the"
f" same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a"
f" directory containing a {PROCESSOR_NAME} file"
) from e

2
mindformers/models/tokenization_utils_base.py
View File

@@ -2084,7 +2084,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
if all(full_file_name is None for full_file_name in resolved_vocab_files.values()):
raise EnvironmentError(
f"Can't load tokenizer for '{pretrained_model_name_or_path}'. If you were trying to load it from "
"'https://openmind.cn/models', make sure you don't have a local directory with "
"'https://modelers.cn/models', make sure you don't have a local directory with "
f"the same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path "
f"to a directory containing all relevant files for a {cls.__name__} tokenizer."
)

10
mindformers/tools/hub/dynamic_module_utils.py
View File

@@ -271,7 +271,7 @@ def get_cached_module_file(
This can be either:
- a string, the *model id* of a pretrained model configuration hosted inside a model repo on
openmind.cn. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced
modelers.cn. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced
under a user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a configuration file saved using the
[`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`.
@@ -293,7 +293,7 @@ def get_cached_module_file(
when running `openmind-cli login` (stored in `~/.openmind`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on openmind.cn, so `revision` can be any
git-based system for storing models and other artifacts on modelers.cn, so `revision` can be any
identifier allowed by git.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, will only try to load the tokenizer configuration from local files.
@@ -457,7 +457,7 @@ def get_class_from_dynamic_module(
This can be either:
- a string, the *model id* of a pretrained model configuration hosted inside a model repo on
openmind.cn. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced
modelers.cn. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced
under a user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a configuration file saved using the
[`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`.
@@ -482,7 +482,7 @@ def get_class_from_dynamic_module(
when running `openmind-cli login` (stored in `~/.openmind`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on openmind.cn, so `revision` can be any
git-based system for storing models and other artifacts on modelers.cn, so `revision` can be any
identifier allowed by git.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, will only try to load the tokenizer configuration from local files.
@@ -491,7 +491,7 @@ def get_class_from_dynamic_module(
code_revision (`str`, *optional*, defaults to `"main"`):
The specific revision to use for the code on the Hub, if the code leaves in a different repository than the
rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for
storing models and other artifacts on openmind.cn, so `revision` can be any identifier allowed by git.
storing models and other artifacts on modelers.cn, so `revision` can be any identifier allowed by git.
<Tip>

27
mindformers/tools/hub/hub.py
View File

@@ -34,9 +34,8 @@ SESSION_ID = uuid4().hex
OPENMIND_DYNAMIC_MODULE_NAME = "openmind_modules"
_is_offline_mode = os.environ.get("TRANSFORMERS_OFFLINE", "0").upper() in ENV_VARS_TRUE_VALUES
_staging_mode = os.environ.get("OPENMIND_CO_STAGING", "NO").upper() in ENV_VARS_TRUE_VALUES
# confirm real default endpoint address
_default_endpoint = "https://hub-ci.openmind.cn" if _staging_mode else "https://openmind.cn"
_default_endpoint = "https://modelers.cn"
OPENMIND_CO_RESOLVE_ENDPOINT = os.environ.get("MDS_ENDPOINT", _default_endpoint)
@@ -236,12 +235,12 @@ def cached_file(
The token to use as HTTP bearer authorization for remote files.
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id,
since we use a git-based system for storing models and other artifacts on openmind.cn,
since we use a git-based system for storing models and other artifacts on modelers.cn,
so `revision` can be any identifier allowed by git.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, will only try to load the tokenizer configuration from local files.
subfolder (`str`, *optional*, defaults to `""`):
In case the relevant files are located inside a subfolder of the model repo on openmind.cn,
In case the relevant files are located inside a subfolder of the model repo on modelers.cn,
you can specify the folder name here.
repo_type (`str`, *optional*):
Specify the repo type (useful when downloading from a space for instance).
@@ -285,7 +284,7 @@ def cached_file(
if _raise_exceptions_for_missing_entries:
raise EnvironmentError(
f"{path_or_repo_id} does not appear to have a file named {full_filename}. Checkout "
f"'https://openmind.cn/{path_or_repo_id}/{revision}' for available files."
f"'https://modelers.cn/{path_or_repo_id}/{revision}' for available files."
)
return None
return resolved_file
@@ -345,11 +344,11 @@ def cached_file(
except GatedRepoError as e:
raise EnvironmentError(
"You are trying to access a gated repo.\nMake sure to request access at "
f"https://openmind.cn/{path_or_repo_id} and pass a token having permission to this repo "
f"https://modelers.cn/{path_or_repo_id} and pass a token having permission to this repo "
"by passing `token=<your_token>`."
) from e
except RepositoryNotFoundError as e:
# replace xxx to openmind.cn
# replace xxx to modelers.cn
raise EnvironmentError(
f"{path_or_repo_id} is not a local folder and is not a valid model identifier "
"listed on 'xxxxxxxxxxxxx'\nIf this is a private repository, make sure to pass a token "
@@ -360,7 +359,7 @@ def cached_file(
raise EnvironmentError(
f"{revision} is not a valid git identifier (branch name, tag name or commit id) that exists "
"for this model name. Check the model page at "
f"'https://openmind.cn/{path_or_repo_id}' for available revisions."
f"'https://modelers.cn/{path_or_repo_id}' for available revisions."
) from e
except LocalEntryNotFoundError as e:
# We try to see if we have a cached version (not up to date):
@@ -383,7 +382,7 @@ def cached_file(
revision = "main"
raise EnvironmentError(
f"{path_or_repo_id} does not appear to have a file named {full_filename}. Checkout "
f"'https://openmind.cn/{path_or_repo_id}/{revision}' for available files."
f"'https://modelers.cn/{path_or_repo_id}/{revision}' for available files."
) from e
except OmHubHTTPError as e:
# First we try to see if we have a cached version (not up to date):
@@ -441,7 +440,7 @@ def get_file_from_repo(
path_or_repo (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a model repo on openmind.cn.
- a string, the *model id* of a model repo on modelers.cn.
- a path to a *directory* potentially containing the file.
filename (`str`):
The name of the file to locate in `path_or_repo`.
@@ -482,7 +481,7 @@ def get_file_from_repo(
Examples:
```python
# Download a tokenizer configuration from openmind.cn and cache.
# Download a tokenizer configuration from modelers.cn and cache.
tokenizer_config = get_file_from_repo("bert-base-uncased", "tokenizer_config.json")
# This model does not have a tokenizer config so the result will be None.
tokenizer_config = get_file_from_repo("xlm-roberta-base", "tokenizer_config.json")
@@ -551,19 +550,19 @@ def has_file(
logger.error(e)
raise EnvironmentError(
f"{path_or_repo} is a gated repository. Make sure to request access at "
f"https://openmind.cn/{path_or_repo} and pass a token having permission to this repo "
f"https://modelers.cn/{path_or_repo} and pass a token having permission to this repo "
"by passing `token=<your_token>`."
) from e
except RepositoryNotFoundError as e:
logger.error(e)
raise EnvironmentError(
f"{path_or_repo} is not a local folder or a valid repository name on 'https://openmind.cn'."
f"{path_or_repo} is not a local folder or a valid repository name on 'https://modelers.cn'."
) from e
except RevisionNotFoundError as e:
logger.error(e)
raise EnvironmentError(
f"{revision} is not a valid git identifier (branch name, tag name or commit id) that exists for this "
f"model name. Check the model page at 'https://openmind.cn/{path_or_repo}' for available revisions."
f"model name. Check the model page at 'https://modelers.cn/{path_or_repo}' for available revisions."
) from e
except requests.HTTPError:
# We return false for EntryNotFoundError (logical) as well as any connection error.