docs/optimization/82_perf_train_cpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9a50159f",
   "metadata": {},
   "source": [
    "# 在 CPU 上高效训练\n",
    "\n",
    "本指南重点介绍如何在 CPU 上高效地训练大型模型。\n",
    "\n",
    "## 使用 IPEX 实现混合精度\n",
    "\n",
    "混合精度是指在模型中同时使用单精度 (fp32) 和半精度 (bf16/fp16) 数据类型来加速训练或推理，同时仍保持大部分单精度的准确性。现代 CPU 如第 3 代、第 4 代和第 5 代 Intel® Xeon® Scalable 处理器原生支持 bf16。第 6 代 Intel® Xeon® Scalable 处理器则支持 bf16 和 fp16。通过启用混合精度训练，你可以直接获得更好的性能提升。\n",
    "\n",
    "为了进一步提高训练性能，你可以使用 Intel® Extension for PyTorch (IPEX)。IPEX 是基于 PyTorch 构建的一个库，它增加了对 Intel® 高级向量扩展 512 矢量神经网络指令 (Intel® AVX512-VNNI) 和 Intel® 高级矩阵扩展 (Intel® AMX) 的支持，从而在 Intel CPU 上提供额外的性能提升。然而，仅支持 AVX2 的 CPU（例如，AMD 或较旧的 Intel CPU）在 IPEX 下不一定有更好性能。\n",
    "\n",
    "自 PyTorch 1.10 版本起，CPU 后端已经启用了自动混合精度 (AMP)。IPEX 也支持 CPU 上的 bf16/fp16 操作优化，并部分合并到主 PyTorch 分支。使用 IPEX AMP 可以获得更好的性能和用户体验。\n",
    "\n",
    "了解更多详细信息，请参阅 [自动混合精度](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/amp.html)。\n",
    "\n",
    "### IPEX 安装\n",
    "\n",
    "IPEX 的发布版本跟随 PyTorch。通过 pip 安装：\n",
    "\n",
    "| PyTorch 版本 | IPEX 版本 |\n",
    "| :---: | :---: |\n",
    "| 2.5.0 | 2.5.0+cpu |\n",
    "| 2.4.0 | 2.4.0+cpu |\n",
    "| 2.3.0 | 2.3.0+cpu |\n",
    "| 2.2.0 | 2.2.0+cpu |\n",
    "\n",
    "请运行 `pip list | grep torch` 获取你的 `pytorch_version`，以便获取相应的 `IPEX version_name`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "077bf2f6",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install intel_extension_for_pytorch==<version_name> -f https://developer.intel.com/ipex-whl-stable-cpu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31c299c3",
   "metadata": {},
   "source": [
    "\n",
    "如果需要，你可以在 [ipex-whl-stable-cpu](https://developer.intel.com/ipex-whl-stable-cpu) 查看最新版本。\n",
    "\n",
    "了解更多的安装方法，请参阅 [IPEX 安装指南](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html)。\n",
    "\n",
    "### 在 Trainer 中使用\n",
    "\n",
    "要在 Trainer 中启用 IPEX 自动混合精度，用户应在训练命令参数中添加 `use_ipex`、`bf16` 或 `fp16` 以及 `no_cuda`。\n",
    "\n",
    "以下是在 [Transformers 问答任务](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)中使用的示例：\n",
    "\n",
    "- 使用 IPEX 和 BF16 自动混合精度在 CPU 上训练：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc486d36",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/question-answering/run_qa.py \\\n",
    "--model_name_or_path google-bert/bert-base-uncased \\\n",
    "--dataset_name squad \\\n",
    "--do_train \\\n",
    "--do_eval \\\n",
    "--per_device_train_batch_size 12 \\\n",
    "--learning_rate 3e-5 \\\n",
    "--num_train_epochs 2 \\\n",
    "--max_seq_length 384 \\\n",
    "--doc_stride 128 \\\n",
    "--output_dir /tmp/debug_squad/ \\\n",
    "--use_ipex \\\n",
    "--bf16 \\\n",
    "--use_cpu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f71ecadc",
   "metadata": {},
   "source": [
    "\n",
    "如果你想在脚本中启用 `use_ipex` 和 `bf16`，可以像这样在 `TrainingArguments` 中添加这些参数：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc603800",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir=args.output_path,\n",
    "    bf16=True,\n",
    "    use_ipex=True,\n",
    "    use_cpu=True,\n",
    "    **kwargs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0b73829",
   "metadata": {},
   "source": [
    "\n",
    "### 实践示例\n",
    "\n",
    "博客：[使用 Intel Sapphire Rapids 加速 PyTorch Transformers](https://huggingface.co/blog/intel-sapphire-rapids)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}