docs/tutorials/6_train_with_script.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用脚本进行训练\n",
    "\n",
    "除了 🤗 Transformers [notebooks](https://huggingface.co/docs/transformers/main/en/notebooks)，还有示例脚本演示了如何使用 [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch)、[TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow)或[JAX/Flax](https://github.com/huggingface/transformers/tree/main/examples/flax)训练模型以解决特定任务。\n",
    "\n",
    "你还可以在这些示例中找到我们在[研究项目](https://github.com/huggingface/transformers/tree/main/examples/research_projects)和[遗留示例](https://github.com/huggingface/transformers/tree/main/examples/legacy)中使用过的脚本，这些脚本主要是由社区贡献的。这些脚本已不再被积极维护，可能与🤗 Transformers 库的最新版本不兼容，需要使用指定版本的🤗 Transformers 。\n",
    "\n",
    "示例脚本可能无法在初始配置下直接解决每个问题，你可能需要根据要解决的问题调整脚本。为了帮助你，大多数脚本都完全暴露了数据预处理的方式，允许你根据需求对其进行编辑。\n",
    "\n",
    "对于你想在示例脚本中实现的任何功能，都可以在论坛或issue上讨论，然后再提交 Pull Request。我们欢迎修复错误，但不太可能会合并那些帮助添加了更多功能的 Pull Request，因为这会降低可读性。\n",
    "\n",
    "本指南将向你展示如何在 [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) 和 [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) 中运行摘要训练脚本的示例。除非有特别说明，否则所有示例都可以在两个框架中工作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## 设置\n",
    "\n",
    "要成功运行示例脚本的最新版本，你必须在新的虚拟环境中，选择从源代码安装 🤗 Transformers："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "git clone https://github.com/huggingface/transformers\n",
    "cd transformers\n",
    "pip install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后切换到你 clone 的 🤗 Transformers 库到特定的版本，例如v3.5.1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "git checkout tags/v3.5.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在安装了正确的🤗 Transformers 库版本之后，进入你选择的版本的 `example` 文件夹并安装示例要求的环境："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 运行脚本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pytorch\n",
    "\n",
    "示例脚本从🤗 Datasets 库下载并预处理数据集。然后，脚本通过 Trainer 使用支持摘要任务的架构对数据集进行微调。\n",
    "\n",
    "以下示例展示了如何在[CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail)数据集上微调[T5-small](https://huggingface.co/google-t5/t5-small)。\n",
    "\n",
    "**由于 T5 模型是通过样本内容的前缀来区分不同的任务（例如翻译、问答、摘要等），因此在示例中需要设置一个额外的参数 `--source_prefix \"summarize: \"`，通过这个前缀提示 T5 这是一个摘要任务。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow\n",
    "\n",
    "示例脚本从🤗 Datasets 库下载并预处理数据集。然后，脚本通过 Keras 使用支持摘要任务的架构对数据集进行微调。\n",
    "\n",
    "以下示例展示了如何在[CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail)数据集上微调[T5-small](https://huggingface.co/google-t5/t5-small)。\n",
    "\n",
    "由于 T5 模型的训练方式，它也需要设置一个额外的参数 `source_prefix`，这个提示让 T5 知道这是一个摘要任务。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/tensorflow/summarization/run_summarization.py  \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization  \\\n",
    "    --per_device_train_batch_size 8 \\\n",
    "    --per_device_eval_batch_size 16 \\\n",
    "    --num_train_epochs 3 \\\n",
    "    --do_train \\\n",
    "    --do_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分布式训练和混合精度\n",
    "\n",
    "[Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) 支持分布式训练和混合精度，这意味着你也可以在脚本中使用它。要启用这两个功能，可以做如下设置：\n",
    "\n",
    "- 添加参数 `fp16`，启用混合精度。\n",
    "- 添加参数 `nproc_per_node`，设置使用的GPU数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "torchrun \\\n",
    "    --nproc_per_node 8 pytorch/summarization/run_summarization.py \\\n",
    "    --fp16 \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TensorFlow 脚本使用 `MirroredStrategy` 进行分布式训练，所以无需在训练脚本中添加任何其他参数。如果可用，TensorFlow 脚本将默认使用多个GPU。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在TPU上运行脚本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch\n",
    "\n",
    "张量处理单元`TPUs`是专门设计用于加速性能的。PyTorch使用`XLA`深度学习编译器来支持 TPU（更多细节请参见这里）。\n",
    "\n",
    "要使用 TPU，需要启动`xla_spawn.py`脚本并使用参数`num_cores`设置要使用的 TPU 核心数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python xla_spawn.py --num_cores 8 \\\n",
    "    summarization/run_summarization.py \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow\n",
    "\n",
    "张量处理单元（TPUs）是专门设计用于加速性能的。TensorFlow 脚本使用 `TPUStrategy` 在 TPU 上进行训练。\n",
    "\n",
    "要使用TPU，需要将要使用的 TPU 资源的名称传递给 `tpu` 参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python run_summarization.py  \\\n",
    "    --tpu name_of_tpu_resource \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --output_dir /tmp/tst-summarization  \\\n",
    "    --per_device_train_batch_size 8 \\\n",
    "    --per_device_eval_batch_size 16 \\\n",
    "    --num_train_epochs 3 \\\n",
    "    --do_train \\\n",
    "    --do_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于🤗 Accelerate运行脚本\n",
    "\n",
    "🤗 [Accelerate](https://huggingface.co/docs/accelerate/index) 是一个仅支持 PyTorch 的库，它会自动检测可用的硬件资源（包括 CPU、GPU 和 TPU）并相应地配置训练环境，对外提供了一个统一的训练接口，让开发者可以在不同硬件上以相同的方式启动训练过程，同时保持对 PyTorch 训练循环的完全可见性。\n",
    "\n",
    "通过 Accelerate，你可以用更少的代码实现复杂的分布式训练设置。如果你还没有安装 🤗 Accelerate，请确保你已经安装了它：\n",
    "\n",
    "**注意：由于 Accelerate 正在快速发展，因此必须安装 git 版本的 accelerate 来运行脚本。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install git+https://github.com/huggingface/accelerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你需要使用`run_summarization_no_trainer.py`脚本，而不是`run_summarization.py`脚本。\n",
    "\n",
    "🤗 Accelerate 支持的脚本需要在文件夹中有一个`task_no_trainer.py`文件。首先运行以下命令来创建和保存配置文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "accelerate config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检测你的设置以确保配置正确："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "accelerate test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在你可以开始训练模型了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "accelerate launch run_summarization_no_trainer.py \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir ~/tmp/tst-summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用自定义数据集\n",
    "\n",
    "摘要脚本支持自定义数据集，只要它们是 CSV 或 JSON Line 文件。当你使用自己的数据集时，需要指定一些额外的参数：\n",
    "\n",
    "- `train_file` 和 `validation_file` 分别是指定你的训练和验证文件的路径。\n",
    "- `text_column` 是指定你的数据集中包含输入文本的列名。具体来说，它指定了数据集中哪一列的文本数据将被用作模型的输入，即指定需要进行摘要的原始文本。\n",
    "- `summary_column` 是指定你的数据集中包含目标摘要文本的列名。具体来说，它指定了数据集中哪一列的文本数据被用作模型的输出目标，即你期望模型生成的摘要文本。\n",
    "\n",
    "使用了自定义数据集的摘要脚本看起来是这样的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --train_file path_to_csv_or_jsonlines_file \\\n",
    "    --validation_file path_to_csv_or_jsonlines_file \\\n",
    "    --text_column text_column_name \\\n",
    "    --summary_column summary_column_name \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --overwrite_output_dir \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试脚本\n",
    "\n",
    "通常，在提交整个数据集之前，最好先在较少的数据集上运行脚本，以确保一切按预期工作。因为处理完整的数据集可能需要花费几个小时。可以添加以下三个参数来截断数据集：\n",
    "\n",
    "- max_train_samples：控制训练数据集的最大样本数。\n",
    "- max_eval_samples：控制验证数据集的最大样本数。\n",
    "- max_predict_samples：控制预测数据集的最大样本数。\n",
    "\n",
    "通过合理地设置这些参数，你可以在处理大型数据集之前，先在较小的数据集上快速验证和调试你的脚本，确保一切按预期工作，从而节省时间和资源。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py \\\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --max_train_samples 50 \\\n",
    "    --max_eval_samples 50 \\\n",
    "    --max_predict_samples 50 \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并非所有示例脚本都支持`max_predict_samples`参数。如果你不确定你的脚本是否支持此参数，可以使用 -h 参数先进行检查："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "examples/pytorch/summarization/run_summarization.py -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 从checkpoint恢复训练 \n",
    "\n",
    "另一个方式是从之前的 checkpoint 恢复训练。确保在训练中断时，可以从之前停止的地方继续进行，而无需重新开始训练。\n",
    "\n",
    "有两种方法可以从 checkpoint 恢复训练：\n",
    "\n",
    "1. 第一种方法是设置参数`output_dir`，从存储在 output_dir 中的最新的 checkpoint 恢复训练。在这种情况下，你应该删除`overwrite_output_dir`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --output_dir previous_output_dir \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 第二种方法设置参数`resume_from_checkpoint`从指定的 checkpoint 文件夹恢复训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --resume_from_checkpoint path_to_specific_checkpoint \\\n",
    "    --predict_with_generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分享模型\n",
    "\n",
    "所有的脚本都可以将你的最终模型上传到 [Model Hub](https://huggingface.co/models)。在开始之前，请确保你已登录 Hugging Face："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "huggingface-cli login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在脚本中添加参数 `push_to_hub`。这个参数会默认创建一个带有你的 Hugging Face用户名 和 output_dir 中指定的文件夹的名称的仓库。\n",
    "\n",
    "想要给你的仓库指定名称，可以添加参数 `push_to_hub_model_id` 来设置。该仓库将在你的命名空间下列出。\n",
    "\n",
    "以下示例展示了如何上传具有指定的仓库名称的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python examples/pytorch/summarization/run_summarization.py\n",
    "    --model_name_or_path google-t5/t5-small \\\n",
    "    --do_train \\\n",
    "    --do_eval \\\n",
    "    --dataset_name cnn_dailymail \\\n",
    "    --dataset_config \"3.0.0\" \\\n",
    "    --source_prefix \"summarize: \" \\\n",
    "    --push_to_hub \\\n",
    "    --push_to_hub_model_id finetuned-t5-cnn_dailymail \\\n",
    "    --output_dir /tmp/tst-summarization \\\n",
    "    --per_device_train_batch_size=4 \\\n",
    "    --per_device_eval_batch_size=4 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --predict_with_generate"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}