docs/guide/29_masked_language_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4743f806",
   "metadata": {},
   "source": [
    "# 遮蔽语言模型\n",
    "\n",
    "遮蔽语言模型预测序列中的遮蔽标记，并且模型可以双向关注标记。这意味着模型可以完全访问左右两边的标记。遮蔽语言模型非常适合需要对整个序列有良好上下文理解的任务。BERT 就是一个遮蔽语言模型的例子。\n",
    "\n",
    "本指南将向你展示如何：\n",
    "\n",
    "1. 在 [ELI5](https://huggingface.co/datasets/eli5) 数据集的 [r/askscience](https://www.reddit.com/r/askscience/) 子集上微调 [DistilRoBERTa](https://huggingface.co/distilbert/distilroberta-base)。\n",
    "2. 使用你的微调模型进行推理。\n",
    "\n",
    "要查看与此任务兼容的所有架构和检查点，我们推荐查看[任务页面](https://huggingface.co/tasks/fill-mask)。\n",
    "\n",
    "开始之前，请确保你已安装所有必要的库：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df0b703c",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install transformers datasets evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8326670c",
   "metadata": {},
   "source": [
    "\n",
    "我们鼓励你登录你的 Hugging Face 账户，这样你就可以上传并与社区分享你的模型。当提示时，输入你的 token 以登录：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "769eefdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98aed161",
   "metadata": {},
   "source": [
    "\n",
    "## 加载 ELI5 数据集\n",
    "\n",
    "首先使用 🤗 Datasets 库加载 [ELI5-Category](https://huggingface.co/datasets/eli5_category) 数据集的前 5000 个示例。这将给你一个机会进行实验并确保一切工作正常，然后再花更多时间在完整数据集上训练。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac1fbfda",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "eli5 = load_dataset(\"eli5_category\", split=\"train[:5000]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ea1d44",
   "metadata": {},
   "source": [
    "\n",
    "使用 [train_test_split](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.train_test_split) 方法将数据集的 `train` 分割成训练集和测试集：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfedcb69",
   "metadata": {},
   "outputs": [],
   "source": [
    "eli5 = eli5.train_test_split(test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1945c957",
   "metadata": {},
   "source": [
    "\n",
    "然后查看一个示例：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17ab5d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "eli5[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ea0a70",
   "metadata": {},
   "source": [
    "\n",
    "虽然这看起来很多，但你真正感兴趣的是 `text` 字段。语言建模任务的酷之处在于你不需要标签（也称为无监督任务），因为下一个词就是标签。\n",
    "\n",
    "## 预处理\n",
    "\n",
    "对于遮蔽语言建模，下一步是加载一个 DistilRoBERTa 分词器来处理 `text` 子字段：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59da3b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilroberta-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb5d020",
   "metadata": {},
   "source": [
    "\n",
    "你会从上面的示例中注意到，`text` 字段实际上嵌套在 `answers` 里面。这意味着你需要使用 [`flatten`](https://huggingface.co/docs/datasets/process#flatten) 方法从其嵌套结构中提取 `text` 子字段：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0814597e",
   "metadata": {},
   "outputs": [],
   "source": [
    "eli5 = eli5.flatten()\n",
    "eli5[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d48a54aa",
   "metadata": {},
   "source": [
    "\n",
    "现在每个子字段都是一个单独的列，如 `answers` 前缀所示，`text` 字段现在是一个列表。不要单独对每个句子进行分词，将列表转换为字符串，以便你可以一起对它们进行分词。\n",
    "\n",
    "这是一个预处理函数，用于将每个示例的字符串列表连接起来并分词结果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aee5ea34",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(examples):\n",
    "    return tokenizer([\" \".join(x) for x in examples[\"answers.text\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776a6ade",
   "metadata": {},
   "source": [
    "\n",
    "要在整个数据集上应用这个预处理函数，使用 🤗 Datasets 的 [map](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map) 方法。你可以通过设置 `batched=True` 来加速 `map` 函数，以便一次处理数据集中的多个元素，并使用 `num_proc` 增加进程数。删除你不需要的任何列：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d05c454",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_eli5 = eli5.map(\n",
    "    preprocess_function,\n",
    "    batched=True,\n",
    "    num_proc=4,\n",
    "    remove_columns=eli5[\"train\"].column_names,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cf14407",
   "metadata": {},
   "source": [
    "\n",
    "这个数据集包含了标记序列，但其中一些序列比模型的最大输入长度还要长。\n",
    "\n",
    "现在你可以使用第二个预处理函数来：\n",
    "\n",
    "- 连接所有的序列\n",
    "- 将连接的序列分割成由 `block_size` 定义的更短的块，它应该比模型的最大输入长度短，并且足够短以适应你的 GPU 内存。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed267fc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "block_size = 128\n",
    "\n",
    "def group_texts(examples):\n",
    "    # 连接所有文本。\n",
    "    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n",
    "    total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
    "    # 我们可以添加填充而不是这种丢弃，如果模型支持的话，你可以根据自己的需要定制这部分。\n",
    "    if total_length >= block_size:\n",
    "        total_length = (total_length // block_size) * block_size\n",
    "    # 按 block_size 分割块。\n",
    "    result = {\n",
    "        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n",
    "        for k, t in concatenated_examples.items()\n",
    "    }\n",
    "    return result\n",
    "\n",
    "lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "100af4ed",
   "metadata": {},
   "source": [
    "\n",
    "现在使用 [DataCollatorForLanguageModeling](/docs/transformers/main/en/main_classes/data_collator#transformers.DataCollatorForLanguageModeling) 创建一批示例。在整理过程中动态填充句子到批次中的最长长度，而不是将整个数据集填充到最大长度，这样做更有效。\n",
    "\n",
    "Pytorch\n",
    "\n",
    "使用序列结束标记作为填充标记，并指定 `mlm_probability` 以在每次遍历数据时随机遮蔽标记：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a0a60dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DataCollatorForLanguageModeling\n",
    "\n",
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9f4399",
   "metadata": {},
   "source": [
    "\n",
    "TensorFlow\n",
    "\n",
    "使用序列结束标记作为填充标记，并指定 `mlm_probability` 以在每次遍历数据时随机遮蔽标记：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aaefbb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DataCollatorForLanguageModeling\n",
    "\n",
    "data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors=\"tf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89b61c0a",
   "metadata": {},
   "source": [
    "\n",
    "## 训练\n",
    "\n",
    "Pytorch\n",
    "\n",
    "如果你不熟悉如何使用 [Trainer](/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) 微调模型，请查看[这里](../training#train-with-pytorch-trainer)的基本教程！\n",
    "\n",
    "现在你可以开始训练你的模型了！使用 [AutoModelForMaskedLM](/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForMaskedLM) 加载 DistilRoBERTa：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60888834",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForMaskedLM\n",
    "\n",
    "model = AutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40ad7f35",
   "metadata": {},
   "source": [
    "\n",
    "此时，只剩下三个步骤：\n",
    "\n",
    "1. 在 [TrainingArguments](/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments) 中定义你的训练超参数。唯一必需的参数是 `output_dir`，它指定了保存模型的位置。你可以通过设置 `push_to_hub=True`（需要登录 Hugging Face 才能上传你的模型）将此模型推送到 Hub。\n",
    "2. 将训练参数传递给 [Trainer](/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) 以及模型、数据集和数据整理器。\n",
    "3. 调用 [train()](/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train) 微调你的模型。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eab9be90",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir=\"my_awesome_eli5_mlm_model\",\n",
    "    eval_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,\n",
    "    num_train_epochs=3,\n",
    "    weight_decay=0.01,\n",
    "    push_to_hub=True,\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=lm_dataset[\"train\"],\n",
    "    eval_dataset=lm_dataset[\"test\"],\n",
    "    data_collator=data_collator,\n",
    "    tokenizer=tokenizer,\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0ba992",
   "metadata": {},
   "source": [
    "\n",
    "训练完成后，使用 [evaluate()](/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.evaluate) 方法评估你的模型并获取它的困惑度：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f831d84",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "eval_results = trainer.evaluate()\n",
    "print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4145d85",
   "metadata": {},
   "source": [
    "\n",
    "然后使用 [push_to_hub()](/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.push_to_hub) 方法将你的模型分享到 Hub，这样每个人都可以使用你的模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4abdb958",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.push_to_hub()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a13f64ce",
   "metadata": {},
   "source": [
    "\n",
    "TensorFlow\n",
    "\n",
    "如果你不熟悉如何使用 Keras 微调模型，请查看[这里](../training#train-a-tensorflow-model-with-keras)的基本教程！\n",
    "\n",
    "要微调 TensorFlow 中的模型，首先设置一个优化器函数、学习率计划和一些训练超参数：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7aaa6ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import create_optimizer, AdamWeightDecay\n",
    "\n",
    "optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65083760",
   "metadata": {},
   "source": [
    "\n",
    "然后你可以使用 [TFAutoModelForMaskedLM](/docs/transformers/main/en/model_doc/auto#transformers.TFAutoModelForMaskedLM) 加载 DistilRoBERTa：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c24bc02",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TFAutoModelForMaskedLM\n",
    "\n",
    "model = TFAutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7486dcda",
   "metadata": {},
   "source": [
    "\n",
    "使用 [prepare_tf_dataset()](/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset) 将你的数据集转换为 `tf.data.Dataset` 格式：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bac4227",
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_train_set = model.prepare_tf_dataset(\n",
    "    lm_dataset[\"train\"],\n",
    "    shuffle=True,\n",
    "    batch_size=16,\n",
    "    collate_fn=data_collator,\n",
    ")\n",
    "\n",
    "tf_test_set = model.prepare_tf_dataset(\n",
    "    lm_dataset[\"test\"],\n",
    "    shuffle=False,\n",
    "    batch_size=16,\n",
    "    collate_fn=data_collator,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1582304",
   "metadata": {},
   "source": [
    "\n",
    "使用 [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) 配置模型进行训练。注意，Transformer 模型都有一个默认的任务相关损失函数，所以你不需要指定一个，除非你想：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94eadf11",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "model.compile(optimizer=optimizer)  # No loss argument!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49bab0ae",
   "metadata": {},
   "source": [
    "\n",
    "这可以通过指定在哪里推送你的模型和分词器来完成 [PushToHubCallback](/docs/transformers/main/en/main_classes/keras_callbacks#transformers.PushToHubCallback)：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5ead6b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.keras_callbacks import PushToHubCallback\n",
    "\n",
    "callback = PushToHubCallback(\n",
    "    output_dir=\"my_awesome_eli5_mlm_model\",\n",
    "    tokenizer=tokenizer,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd371a6c",
   "metadata": {},
   "source": [
    "\n",
    "最后，你准备好开始训练你的模型了！用你的训练和验证数据集、训练周期数和你的回调来调用 [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) 微调模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0e083ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8551fb0b",
   "metadata": {},
   "source": [
    "\n",
    "一旦训练完成，你的模型会自动上传到 Hub，这样每个人都可以使用它！\n",
    "\n",
    "有关如何微调遮蔽语言模型的更深入示例，请查看相应的 [PyTorch 笔记本](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb) 或 [TensorFlow 笔记本](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb)。\n",
    "\n",
    "## 推理\n",
    "\n",
    "太好了，现在你已经微调了一个模型，你可以用它进行推理了！\n",
    "\n",
    "想出一些你想让模型填补空白的文本，并使用特殊的 `<mask>` 标记来表示空白：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0852d2f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"The Milky Way is a <mask> galaxy.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "715796c2",
   "metadata": {},
   "source": [
    "\n",
    "尝试你的微调模型进行推理的最简单方法是使用它在 [pipeline()](/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline) 中。为你的模型实例化一个 `pipeline` 进行填充掩码，并传递你的文本给它。如果喜欢，你可以使用 `top_k` 参数来指定返回多少个预测：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb314490",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "mask_filler = pipeline(\"fill-mask\", \"username/my_awesome_eli5_mlm_model\")\n",
    "mask_filler(text, top_k=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cdecad3",
   "metadata": {},
   "source": [
    "\n",
    "Pytorch\n",
    "\n",
    "将文本分词并返回 PyTorch 张量的 `input_ids`。你还需要指定 `<mask>` 标记的位置：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90f6c050",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n",
    "inputs = tokenizer(text, return_tensors=\"pt\")\n",
    "mask_token_index = torch.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)[1]\n",
    "\n",
    "from transformers import AutoModelForMaskedLM\n",
    "\n",
    "model = AutoModelForMaskedLM.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n",
    "logits = model(**inputs).logits\n",
    "mask_token_logits = logits[0, mask_token_index, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6ce274b",
   "metadata": {},
   "source": [
    "\n",
    "然后返回三个可能性最高的遮蔽标记并打印它们：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4103ec77",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()\n",
    "\n",
    "for token in top_3_tokens:\n",
    "    print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a71368",
   "metadata": {},
   "source": [
    "\n",
    "TensorFlow\n",
    "\n",
    "将文本分词并返回 TensorFlow 张量的 `input_ids`。你还需要指定 `<mask>` 标记的位置：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37370105",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n",
    "inputs = tokenizer(text, return_tensors=\"tf\")\n",
    "mask_token_index = tf.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)[0, 1]\n",
    "\n",
    "from transformers import TFAutoModelForMaskedLM\n",
    "\n",
    "model = TFAutoModelForMaskedLM.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n",
    "logits = model(**inputs).logits\n",
    "mask_token_logits = logits[0, mask_token_index, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5bb919",
   "metadata": {},
   "source": [
    "\n",
    "然后返回三个可能性最高的遮蔽标记并打印它们：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b96871d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_3_tokens = tf.math.top_k(mask_token_logits, 3).indices.numpy()\n",
    "\n",
    "for token in top_3_tokens:\n",
    "    print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81999aba",
   "metadata": {},
   "source": [
    "\n",
    "这样，你就可以使用微调后的模型来预测遮蔽词，并得到最有可能的三个填充词。这种方法可以应用于任何需要上下文理解的任务，例如文本生成、问答系统等。\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}