docs/tutorials/4_preprocess_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 预处理数据\n",
    "\n",
    "在数据集上训练模型之前，数据需要被预处理为模型期望的输入格式。无论数据是文本、图像还是音频，它们都需要被转换并组合成批量的张量。\n",
    "\n",
    "🤗 Transformers 提供了一组预处理类来帮助准备数据以供模型使用。在本教程中，你将了解以下内容：\n",
    "1. 对于文本，使用[分词器(Tokenizer)](https://huggingface.co/docs/transformers/v4.44.2/zh/main_classes/tokenizer)能够将文本转换为一系列标记`tokens`，并创建 tokens 的数字表示，将它们组合成张量。\n",
    "2. 对于语音和音频，使用[特征提取器(Feature extractor)](https://huggingface.co/docs/transformers/v4.44.2/zh/main_classes/feature_extractor)能够从音频波形中提取顺序特征并将其转换为张量。\n",
    "3. 对于图像，使用`图像处理器(ImageProcessor)`将图像转换为张量。\n",
    "4. 对于多模态输入，使用[处理器(Processor)](https://huggingface.co/docs/transformers/v4.44.2/zh/main_classes/processors)，其结合了 Tokenizer 和 ImageProcessor 或 Processor。\n",
    "\n",
    "`AutoProcessor` 能够有效的自动选择适用于模型输入的预处理工具，无论你使用的是 `Tokenizer`、`ImageProcessor`、`Feature extractor`还是`Processor`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在开始之前，请安装`🤗 Datasets`，可以加载一些数据集来进行实验："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "pip install datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## 自然语言处理（文本预处理）\n",
    "\n",
    "处理文本数据的主要工具是`分词器(Tokenizer)`，模型所需的任何附加输入都由 Tokenizer 进行添加。\n",
    "\n",
    "Tokenizer 会根据一组规则将文本拆分为 tokens，然后将这些tokens转换为数字，然后转换为张量，成为模型的输入。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**如果你选择使用预训练模型，则需要注意使用与模型关联的预训练的Tokenizer。**这是为了确保文本的拆分方式与预训练的语料库相同，并在预训练期间使用相同的对应关系`标记-索引`（通常称为词汇表`vocab`）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "开始使用`AutoTokenizer.from_pretrained()`方法加载一个预训练的 Tokenizer，并下载预训练模型的词汇表 vocab："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"google-bert/bert-base-cased\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后文本输入给 Tokenizer："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded_input = tokenizer(\"Do not meddle in the affairs of wizards, for they are subtle and quick to anger.\")\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizer 会返回一个包含三个重要对象的字典：\n",
    "\n",
    "- `input_ids`：这是一个整数序列，每个整数代表词汇表中的一个特定token。**模型使用这个序列作为输入来理解和处理文本数据**。\n",
    "- `token_type_ids`：这是一个与 input_ids 相同长度的序列，它表示输入序列中每个 token 的类型。这个字段主要用于**区分不同类型的输入序列**，尤其是在处理由多个文本序列组成的输入时，例如问答系统中的问题和答案，或者文本分类任务中的文本和标签。\n",
    "- `attention_mask`：这个序列表示输入序列中哪些 token 是实际的文本token，哪些是填充（padding）token。在处理批量数据时，**由于不同序列的长度可能不同，通常需要使用填充token来使所有序列长度一致**。`attention_mask` 中的 `1 表示实际的文本token`，而 `0 表示填充token`，这样模型就可以忽略填充token。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过`tokenizer.decode()`可以解码 `input_ids` 得到输入的文本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decode(encoded_input[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从解码得到的文本可知，tokenizer 在句子中添加了两个特殊 token - `[CLS]` 和 `[SEP]`（分类器和分隔符）。\n",
    "\n",
    "**注意！并不是所有模型都需要添加特殊 token，但如果需要，tokenizer 会自动给你添加。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果有多个句子需要进行预处理，可以将它们作为列表传递给 tokenizer："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_sentences = [\n",
    "    \"But what about second breakfast?\",\n",
    "    \"Don't think he knows about second breakfast, Pip.\",\n",
    "    \"What about elevensies?\",\n",
    "]\n",
    "encoded_inputs = tokenizer(batch_sentences)\n",
    "print(encoded_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 填充\n",
    "\n",
    "输入句子的长度并不总是相同的，这是一个问题，因为模型输入的张量需要具有统一的形状。\n",
    "\n",
    "填充是一种解决办法，通过在较短的句子中添加一个特殊的 `padding token`，以确保张量是矩形的。\n",
    "\n",
    "设置参数 `padding=True`，将批次中较短的序列填充到与最长的序列相同的长度，一般用 0 进行填充："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_sentences = [\n",
    "    \"But what about second breakfast?\",\n",
    "    \"Don't think he knows about second breakfast, Pip.\",\n",
    "    \"What about elevensies?\",\n",
    "]\n",
    "encoded_input = tokenizer(batch_sentences, padding=True)\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 截断\n",
    "\n",
    "另一方面，有时候一个序列可能对模型来说太长了。在这种情况下，则需要将序列截断为更短的长度。\n",
    "\n",
    "设置参数 `truncation=True`，将过长的序列截断为模型接受的最大长度，也可以通过 max_length 参数设置截断的最大长度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_sentences = [\n",
    "    \"But what about second breakfast?\",\n",
    "    \"Don't think he knows about second breakfast, Pip.\",\n",
    "    \"What about elevensies?\",\n",
    "]\n",
    "encoded_input = tokenizer(batch_sentences, padding=True, truncation=True)\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看[填充和截断](https://huggingface.co/docs/transformers/main/en/pad_truncation)概念指南，了解更多有关填充和截断参数的信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构建张量\n",
    "\n",
    "最后，`tokenizer` 会返回实际输入到模型的张量。\n",
    "\n",
    "设置参数 `return_tensors=\"pt\"`（对于 PyTorch ）或 `return_tensors=\"tf\"`（对于 TensorFlow ）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PyTorch\n",
    "batch_sentences = [\n",
    "    \"But what about second breakfast?\",\n",
    "    \"Don't think he knows about second breakfast, Pip.\",\n",
    "    \"What about elevensies?\",\n",
    "]\n",
    "encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TensorFlow\n",
    "batch_sentences = [\n",
    "    \"But what about second breakfast?\",\n",
    "    \"Don't think he knows about second breakfast, Pip.\",\n",
    "    \"What about elevensies?\",\n",
    "]\n",
    "encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors=\"tf\")\n",
    "print(encoded_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 音频\n",
    "\n",
    "对于音频任务，需要使用 `feature extractor` 来预处理数据集以供模型使用。\n",
    "\n",
    "`Feature extractor` 能够从原始的音频数据中提取出特征，并将它们转换为张量。\n",
    "\n",
    "加载[MInDS-14](https://huggingface.co/datasets/PolyAI/minds14)数据集（有关如何加载数据集的更多详细信息，请参阅 🤗[Datasets教程](https://huggingface.co/docs/datasets/load_hub)）以了解如何在音频数据集中使用 feature extractor："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, Audio\n",
    "\n",
    "dataset = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")\n",
    "\n",
    "# 访问 audio 列的第一个元素以查看输入。调用 audio 列会自动加载和重新采样音频文件：\n",
    "dataset[0][\"audio\"] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dataset[0][\"audio\"]返回的结果包含了三个对象：\n",
    "- `array` 是一个包含音频样本振幅值的数组，会在必要时重新采为一位数组（1D array），每个元素对应一个音频样本的振幅，采样率决定了每秒钟采集的样本数。\n",
    "- `path` 是音频文件的位置。\n",
    "- `sampling_rate` 是音频的采样率，单位是赫兹（Hz）。常见的采样率有 16000 Hz、22050 Hz、44100 Hz 等。\n",
    "\n",
    "假设有一个采样率为 16000 Hz 的音频信号，持续时间为1秒，那么这个音频信号可以用一个包含16000个元素的1D array来表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，加载一个`feature extractor`以对输入进行标准化和填充。\n",
    "\n",
    "填充文本的理念同样适用于预处理音频数据。当填充文本数据时，会为较短的序列填充 0。在`feature extractor`中，会为较短的数组填充 0 表示静音。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `AutoFeatureExtractor.from_pretrained()` 加载 feature extractor："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoFeatureExtractor\n",
    "\n",
    "feature_extractor = AutoFeatureExtractor.from_pretrained(\"facebook/wav2vec2-base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将音频样本的数组`dataset[0][\"audio\"][\"array\"]`传递给 feature extractor。："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audio_input = [dataset[0][\"audio\"][\"array\"]]\n",
    "feature_extractor(audio_input, sampling_rate=16000) # 建议在 feature extractor 中设置 sampling_rate 参数为音频样本的采样率，以更好地调试可能发生的静音错误"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "就像`tokenizer`一样，可以应用填充或截断来处理批次中不同长度的序列。请查看这两个音频样本的序列长度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset[0][\"audio\"][\"array\"].shape\n",
    "\n",
    "dataset[1][\"audio\"][\"array\"].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个函数来预处理数据集，以使音频样本具有相同的长度。通过指定样本最大的长度`max_length`，`feature extractor` 将填充或截断序列以使其匹配："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(examples):\n",
    "    audio_arrays = [x[\"array\"] for x in examples[\"audio\"]]\n",
    "    inputs = feature_extractor(\n",
    "        audio_arrays,\n",
    "        sampling_rate=16000,\n",
    "        padding=True,\n",
    "        max_length=100000,\n",
    "        truncation=True,\n",
    "    )\n",
    "    return inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将预处理函数`preprocess_function`应用于数据集中的前几个示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_dataset = preprocess_function(dataset[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在样本的长度是相同的，并且与指定的最大长度相匹配。现在可以将经过处理的数据集输入模型了！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_dataset[\"input_values\"][0].shape\n",
    "\n",
    "processed_dataset[\"input_values\"][1].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算机视觉（图像预处理）\n",
    "\n",
    "对于计算机视觉任务，需要使用`image processor`来预处理数据集以供模型使用。\n",
    "\n",
    "图像预处理包括多个步骤，能够将图像转换为模型期望的输入格式。这些步骤包括调整图像大小、标准化、颜色通道校正以及将图像转换为张量等操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "图像预处理通常遵循某种形式的图像增强。图像预处理和图像增强都会改变图像数据，但它们有不同的目的：\n",
    "\n",
    "- `图像增强`可以帮助防止过拟合并增加模型的鲁棒性。你可以在数据增强方面充分发挥创造性，如调整亮度和颜色、裁剪、旋转、调整大小、缩放等，但要注意不要改变图像的含义。\n",
    "- `图像预处理`确保图像与模型预期的输入格式匹配。在微调计算机视觉模型时，必须保证对样本图像进行与模型训练时相同的预处理。**对于图像预处理，请使用与模型相关联的 `ImageProcessor`**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载[food101](https://huggingface.co/datasets/food101)数据集（有关如何加载数据集的更多详细信息，请参阅🤗[Datasets教程](https://huggingface.co/docs/datasets/load_hub)）以了解如何在计算机视觉数据集中使用图像处理器："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为图像数据集相当大，请使用 🤗 Datasets 的 `split` 参数加载少量的训练样本."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset(\"food101\", split=\"train[:100]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用🤗 Datasets 的 `[Image](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=image#datasets.Image)` 功能查看图像："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset[0][\"image\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![vision-preprocess-tutorial](../../resources/images/vision-preprocess-tutorial.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `AutoImageProcessor.from_pretrained()` 加载 image processor："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoImageProcessor\n",
    "\n",
    "image_processor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，让我们进行图像增强。你可以使用任何你喜欢的图像增强库。\n",
    "\n",
    "如果你有兴趣使用其他的图像增强库，可以参考一下[Albumentations](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_albumentations.ipynb)或[Kornia notebooks](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_kornia.ipynb)中的示例。\n",
    "\n",
    "在本教程中，我们将使用`torchvision`的[transforms](https://pytorch.org/vision/stable/transforms.html)库来进行图像增强。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 在这里，我们使用[Compose](https://pytorch.org/vision/master/generated/torchvision.transforms.Compose.html)将[RandomResizedCrop](https://pytorch.org/vision/main/generated/torchvision.transforms.RandomResizedCrop.html)和[ColorJitter](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html)变换连接在一起。**请注意，对于调整大小，我们可以从image_processor中获取图像尺寸要求。**对于一些模型，需要定义精确的高度和宽度，而对于其他模型则只需定义`shortest_edge`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose\n",
    "except ImportError:\n",
    "    raise ImportError(\"'torchvision' could not be resolved. Please install it by 'pip install torchvision'.\")\n",
    "\n",
    "size = (\n",
    "    image_processor.size[\"shortest_edge\"]\n",
    "    if \"shortest_edge\" in image_processor.size\n",
    "    else (image_processor.size[\"height\"], image_processor.size[\"width\"]) # 如果不包含 shortest_edge 键，则 size 变量被设置为图像的原始高度和宽度。\n",
    ")\n",
    "\n",
    "_transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`shortest_edge`：指的是图像的高度和宽度中的较小值，以确保处理后的图像具有一致的最短边长。例如，对于一个尺寸为 800x600 的图像，即shortest_edge=600。\n",
    "\n",
    "`image_processor.size`：是一个字典，包含了图像处理的相关尺寸信息。可能包含 `shortest_edge`、`height` 和 `width` 等健。\n",
    "\n",
    "`RandomResizedCrop(size)`：\n",
    "- 这个变换会随机裁剪图像并调整到指定的 size。\n",
    "- 如果 size 是一个整数，则表示裁剪后的图像的最短边长。\n",
    "- 如果 size 是一个元组 (height, width)，则表示裁剪后的图像的目标尺寸。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 模型接受 `pixel_values` 作为输入。`ImageProcessor` 可以进行图像标准化，并生成张量。创建一个函数，将图像增强和图像预处理的步骤组合起来处理批量图像，并生成 `pixel_values`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transforms(examples):\n",
    "    images = [_transforms(img.convert(\"RGB\")) for img in examples[\"image\"]]\n",
    "    examples[\"pixel_values\"] = image_processor(images, do_resize=False, return_tensors=\"pt\")[\"pixel_values\"]\n",
    "    return examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认情况下`ImageProcessor`会调整图像的大小。因为我们已经在图像增强转换中调整了图像的大小(`_transforms` 中已经包含了尺寸调整的步骤（如`RandomResizedCrop`）)，所以不需要 `image_processor` 再次调整尺寸，通过设置参数 `do_resize=False`可以保持图像的原始尺寸。\n",
    "\n",
    "如果想要将图像标准化步骤为图像增强的一部分，请使用`image_processor.image_mean`和`image_processor.image_std`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 然后使用 🤗 Datasets 的[set_transform](https://huggingface.co/docs/datasets/process#format-transform)在运行时对图片应用`transforms`进行变换："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.set_transform(transforms)\n",
    "\n",
    "# 当你访问图像时，可以看到`image processor`已添加了`pixel_values（像素值）`。\n",
    "dataset[0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 现在可以将经过处理的数据集输入给模型了！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "img = dataset[0][\"pixel_values\"]\n",
    "plt.imshow(img.permute(1, 2, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是在应用变换后的图像样子。图像已被随机裁剪，并使其颜色属性发生了变化。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![preprocessed_image](../../resources/images/preprocessed_image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于诸如目标检测、语义分割、实例分割和全景分割等任务，`ImageProcessor` 都提供了训练后处理方法。**这些方法能够将模型的原始输出转换为有意义的预测结果，如边界框或分割地图。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 填充\n",
    "\n",
    "在某些情况下，例如，在微调[DETR](https://huggingface.co/docs/transformers/main/en/model_doc/detr)时，模型在训练阶段应用了尺度增强，这可能导致批处理中的图像大小不同。\n",
    "\n",
    "我们可以定义一个函数`collate_fn`，用于在数据加载过程中对每个批次（batch）的数据进行整理和预处理。\n",
    "\n",
    "在处理图像和文本数据时，`collate_fn` 通常设定为将一个批次中的多个样本整理成一个统一的格式，以便于模型训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collate_fn(batch):\n",
    "    # 1. 提取像素值：从每个样本中提取 pixel_values，这是一个包含图像数据的列表。\n",
    "    pixel_values = [item[\"pixel_values\"] for item in batch]\n",
    "    # 2. 填充操作：使用 DetrImageProcessor.pad() 方法对图像数据进行填充，确保所有图像具有相同的尺寸，返回 PyTorch 张量。\n",
    "    encoding = image_processor.pad(pixel_values, return_tensors=\"pt\")\n",
    "    # 3. 提取标签：从每个样本中提取 labels，这是一个包含标签的列表。\n",
    "    labels = [item[\"labels\"] for item in batch]\n",
    "    # 4. 创建批次字典：将填充后的像素值、像素掩码和标签整理成一个字典，便于后续的模型输入。\n",
    "    batch = {}\n",
    "    batch[\"pixel_values\"] = encoding[\"pixel_values\"]\n",
    "    batch[\"pixel_mask\"] = encoding[\"pixel_mask\"]\n",
    "    batch[\"labels\"] = labels\n",
    "    return batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为什么需要 `collate_fn` \n",
    "\n",
    "- 批处理一致性：在深度学习中，模型通常以批次为单位进行训练，通过 collate_fn 确保每个批次的数据格式是一致的。\n",
    "- 数据填充：不同样本的图像尺寸可能不同，通过 collate_fn 中的填充操作可以使它们具有相同的尺寸，便于批处理。\n",
    "- 掩码生成：掩码可以帮助模型区分实际数据和填充数据，避免模型在训练过程中受到填充数据的干扰。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多模态\n",
    "\n",
    "对于涉及多模态输入的任务，你需要 processor 来为模型准备数据集。\n",
    "\n",
    "`Processor` 将两个处理对象组合在一起（例如tokenizer和feature extractor）。\n",
    "\n",
    "加载[LJ Speech](https://huggingface.co/datasets/keithito/lj_speech)数据集，有关如何加载数据集的更多详细信息，请参阅🤗 [Datasets](https://huggingface.co/docs/datasets/load_hub) 教程，以了解如何使用 processor 进行自动语音识别（ASR）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "lj_speech = load_dataset(\"lj_speech\", split=\"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于`ASR`（自动语音识别），主要关注`Audio`和`Text`，因此可以删除其他列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lj_speech = lj_speech.map(remove_columns=[\"file\", \"id\", \"normalized_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看`Audio`和`Text`列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lj_speech[0][\"audio\"]\n",
    "\n",
    "lj_speech[0][\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，你应始终重新采样音频数据集的采样率，与用于预训练模型数据集的采样率保持一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lj_speech = lj_speech.cast_column(\"audio\", Audio(sampling_rate=16_000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用`AutoProcessor.from_pretrained()`加载一个 Processor："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoProcessor\n",
    "\n",
    "processor = AutoProcessor.from_pretrained(\"facebook/wav2vec2-base-960h\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 定义一个函数`prepare_dataset`，用于将包含在 `array` 中的音频数据处理为 `input_values`，并将 `text` 标记为 `labels`。这些将是输入模型的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_dataset(example):\n",
    "    audio = example[\"audio\"]\n",
    "    example.update(processor(audio=audio[\"array\"], text=example[\"text\"], sampling_rate=16000))\n",
    "    return example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 将 `prepare_dataset` 应用于一个示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prepare_dataset(lj_speech[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Processor` 现在已经添加了 `input_values` 和 `labels`，并且采样率也正确地降低为16kHz。现在可以将处理后的数据集输入给模型了！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}