YATO-Handbook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d222ebb3",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# YATO Handbook\n",
    "\n",
    "**YATO**, an open-source Python library for text analysis. In particular, **YATO** focus on sequence labeling and sequence classification tasks, including extensive fundamental NLP tasks such as par-of-speech tagging, chunking, NER, CCG supertagging, sentiment analysis, and sentence classification. **YATO** support both designing specific RNN-based and Transformer-based through user-friendly configuration and integrating the SOTA pre-trained language models such as BERT. \n",
    "\n",
    "\n",
    "## Data Preparation \n",
    "**YATO** supports sequence labeling and sequence classification tasks.   \n",
    "**YATO** offers an official function to convert sequence labeling dataset style.**Location:ncrf.utils.tagSchemeConverter**  \n",
    "NER Dataset Style: BIO, BIOES, BMES...   \n",
    "Sequence Classification: Text ||| Label  \n",
    "\n",
    "## Configuration Preparation  \n",
    "You can specify the model, optimizer, and decoding through the configuration file.   \n",
    "### Dataloader  \n",
    "train_dir=the path of train file    \n",
    "dev_dir=the path of validation file   \n",
    "test_dir=the path of test file    \n",
    "model_dir=the path to save model weights  \n",
    "dset_dir=the path of configuration encode file    \n",
    "### Model\n",
    "use_crf=True/False     \n",
    "use_char=True/False     \n",
    "char_seq_feature=GRU/LSTM/CNN/False     \n",
    "use_word_seq=True/False     \n",
    "use_word_emb=True/False     \n",
    "word_emb_dir=The path of word embedding file    \n",
    "word_seq_feature=GRU/LSTM/CNN/FeedFowrd/False   \n",
    "low_level_transformer=pretrain language model from huggingface  \n",
    "low_level_transformer_finetune=True/False  \n",
    "high_level_transformer=pretrain language model from huggingface  \n",
    "high_level_transformer_finetune=True/False      \n",
    "cnn_layer=layer number     \n",
    "char_hidden_dim=dimension number      \n",
    "hidden_dim=dimension number     \n",
    "lstm_layer=layer number      \n",
    "bilstm=True/False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471c6328",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Hyperparameters       \n",
    "sentence_classification=True/False        \n",
    "status=train/decode         \n",
    "dropout=Dropout Rate         \n",
    "optimizer=SGD/Adagrad/adadelta/rmsprop/adam/adamw    \n",
    "iteration=epoch number         \n",
    "batch_size=batch size           \n",
    "learning_rate=learning rate         \n",
    "gpu=True/False         \n",
    "device=cuda:0        \n",
    "scheduler=get_linear_schedule_with_warmup/get_cosine_schedule_with_warmup            \n",
    "warmup_step_rate=warmup steo rate         "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c0a3c0",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f606d5c8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MODEL: train\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "Start   Sequence   Laebling   task...\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "DATA SUMMARY START:\n",
      " I/O:\n",
      "     Train  file directory: /home/zqwang/YATO-Github/sample_data/ner.train.bmes\n",
      "     Dev    file directory: /home/zqwang/YATO-Github/sample_data/ner.dev.bmes\n",
      "     Test   file directory: /home/zqwang/YATO-Github/sample_data/ner.test.bmes\n",
      "     Raw    file directory: None\n",
      "     Dset   file directory: /data_8t/YATO_MODEL/bert_base_sample.dset\n",
      "     Model  file directory: /data_8t/YATO_MODEL/\n",
      "     Loadmodel   directory: None\n",
      "     Decode file directory: None\n",
      "++++++++++++++++++++++++++++++++++++++++\n",
      "Data and Settings:\n",
      "     Tag          scheme: BIOES\n",
      "     Split         token:  ||| \n",
      "     MAX SENTENCE LENGTH: 512\n",
      "     MAX   WORD   LENGTH: -1\n",
      "     Number   normalized: False\n",
      "     Word         cutoff: 0\n",
      "     Train instance number: 484\n",
      "     Dev   instance number: 112\n",
      "     Test  instance number: 186\n",
      "     Raw   instance number: 0\n",
      "     Word  alphabet size: 3429\n",
      "     Char  alphabet size: 80\n",
      "     Label alphabet size: 18\n",
      "     Label alphabet content: {'instance2index': {'S-ORG': 1, 'O': 2, 'S-MISC': 3, 'B-PER': 4, 'E-PER': 5, 'S-LOC': 6, 'B-ORG': 7, 'E-ORG': 8, 'I-PER': 9, 'S-PER': 10, 'B-MISC': 11, 'I-MISC': 12, 'E-MISC': 13, 'I-ORG': 14, 'B-LOC': 15, 'E-LOC': 16, 'I-LOC': 17}, 'instances': ['S-ORG', 'O', 'S-MISC', 'B-PER', 'E-PER', 'S-LOC', 'B-ORG', 'E-ORG', 'I-PER', 'S-PER', 'B-MISC', 'I-MISC', 'E-MISC', 'I-ORG', 'B-LOC', 'E-LOC', 'I-LOC']}\n",
      "     FEATURE num: 0\n",
      " ++++++++++++++++++++++++++++++++++++++++\n",
      " Model Network:\n",
      "     Model        use_crf: False\n",
      "     Model low level transformer: None; Finetune: False\n",
      "     Model high level transformer: bert-base-uncased; Finetune: True\n",
      "     Classification_Head: False; Classification_Head Activation: tahn\n",
      "     Classifier         dropout: 0.1\n",
      " ++++++++++++++++++++++++++++++++++++++++\n",
      " Training:\n",
      "     Optimizer: AdamW\n",
      "     Scheduler: get_cosine_schedule_with_warmup\n",
      "     warmup_step_rate: 0.050000\n",
      "     Iteration: 10\n",
      "     BatchSize: 16\n",
      "     Average  batch   loss: False\n",
      " ++++++++++++++++++++++++++++++++++++++++\n",
      " Hyperparameters:\n",
      "     Hyper              lr: 3e-05\n",
      "     Hyper         HP_clip: None\n",
      "     Hyper        momentum: 0\n",
      "     Hyper              l2: 1e-08\n",
      "     Hyper      hidden_dim: 768\n",
      "     Hyper         dropout: 0.3\n",
      "     Hyper             GPU: True\n",
      "     Hyper          device: cuda:0\n",
      "DATA SUMMARY END.\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training model...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "build sequence labeling network...\n",
      "use_char:  False\n",
      "use crf:  False\n",
      "build word representation...\n",
      "Loading transformer... model: bert-base-uncased\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " ++++++++++++++++++++++++++++++++++++++++\n",
      "BertConfig {\n",
      "  \"_name_or_path\": \"bert-base-uncased\",\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.18.0\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      " ++++++++++++++++++++++++++++++++++++++++\n",
      "Epoch: 0/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 29.92it/s]\n",
      "Dev: time: 0.27s, speed: 415.10st/s; acc: 0.8011, p: -1.0000, r: 0.0000, f: -1.0000 \n",
      "INFO:test:Dev: time: 0.27s, speed: 415.10st/s; acc: 0.8011, p: -1.0000, r: 0.0000, f: -1.0000 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1168  All token =  1458  acc =  0.8010973936899863\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.93it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.8712_p-1.0000_r0.0000_f-1.0000.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.8712_p-1.0000_r0.0000_f-1.0000.pth\n",
      "Test: time: 0.58s, speed: 323.18st/s; acc: 0.8712, p: -1.0000, r: 0.0000, f: -1.0000 \n",
      "INFO:test:Test: time: 0.58s, speed: 323.18st/s; acc: 0.8712, p: -1.0000, r: 0.0000, f: -1.0000 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3145  All token =  3610  acc =  0.871191135734072\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 27.35it/s]\n",
      "Dev: time: 0.30s, speed: 380.27st/s; acc: 0.8567, p: 0.2512, r: 0.2476, f: 0.2494 \n",
      "INFO:test:Dev: time: 0.30s, speed: 380.27st/s; acc: 0.8567, p: 0.2512, r: 0.2476, f: 0.2494 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1249  All token =  1458  acc =  0.8566529492455418\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.71it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9080_p0.3528_r0.3671_f0.3598.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9080_p0.3528_r0.3671_f0.3598.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9080_p0.3528_r0.3671_f0.3598.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9080_p0.3528_r0.3671_f0.3598.pth\n",
      "Test: time: 0.59s, speed: 319.78st/s; acc: 0.9080, p: 0.3528, r: 0.3671, f: 0.3598 \n",
      "INFO:test:Test: time: 0.59s, speed: 319.78st/s; acc: 0.9080, p: 0.3528, r: 0.3671, f: 0.3598 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3278  All token =  3610  acc =  0.9080332409972299\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 2/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 30.53it/s]\n",
      "Dev: time: 0.27s, speed: 423.79st/s; acc: 0.8937, p: 0.4429, r: 0.4429, f: 0.4429 \n",
      "INFO:test:Dev: time: 0.27s, speed: 423.79st/s; acc: 0.8937, p: 0.4429, r: 0.4429, f: 0.4429 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1303  All token =  1458  acc =  0.8936899862825789\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 19.58it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9238_p0.4928_r0.4913_f0.4920.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9238_p0.4928_r0.4913_f0.4920.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9238_p0.4928_r0.4913_f0.4920.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9238_p0.4928_r0.4913_f0.4920.pth\n",
      "Test: time: 0.62s, speed: 302.13st/s; acc: 0.9238, p: 0.4928, r: 0.4913, f: 0.4920 \n",
      "INFO:test:Test: time: 0.62s, speed: 302.13st/s; acc: 0.9238, p: 0.4928, r: 0.4913, f: 0.4920 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3335  All token =  3610  acc =  0.9238227146814404\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 3/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 23.77it/s]\n",
      "Dev: time: 0.34s, speed: 330.74st/s; acc: 0.9170, p: 0.6029, r: 0.5857, f: 0.5942 \n",
      "INFO:test:Dev: time: 0.34s, speed: 330.74st/s; acc: 0.9170, p: 0.6029, r: 0.5857, f: 0.5942 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1337  All token =  1458  acc =  0.9170096021947873\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.14it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9427_p0.6401_r0.6272_f0.6336.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9427_p0.6401_r0.6272_f0.6336.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9427_p0.6401_r0.6272_f0.6336.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9427_p0.6401_r0.6272_f0.6336.pth\n",
      "Test: time: 0.60s, speed: 311.06st/s; acc: 0.9427, p: 0.6401, r: 0.6272, f: 0.6336 \n",
      "INFO:test:Test: time: 0.60s, speed: 311.06st/s; acc: 0.9427, p: 0.6401, r: 0.6272, f: 0.6336 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3403  All token =  3610  acc =  0.9426592797783934\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 4/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 28.11it/s]\n",
      "Dev: time: 0.29s, speed: 390.08st/s; acc: 0.9472, p: 0.7333, r: 0.7857, f: 0.7586 \n",
      "INFO:test:Dev: time: 0.29s, speed: 390.08st/s; acc: 0.9472, p: 0.7333, r: 0.7857, f: 0.7586 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1381  All token =  1458  acc =  0.9471879286694102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.56it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9476_p0.6374_r0.6705_f0.6535.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9476_p0.6374_r0.6705_f0.6535.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9476_p0.6374_r0.6705_f0.6535.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9476_p0.6374_r0.6705_f0.6535.pth\n",
      "Test: time: 0.59s, speed: 316.66st/s; acc: 0.9476, p: 0.6374, r: 0.6705, f: 0.6535 \n",
      "INFO:test:Test: time: 0.59s, speed: 316.66st/s; acc: 0.9476, p: 0.6374, r: 0.6705, f: 0.6535 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3421  All token =  3610  acc =  0.9476454293628809\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 5/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 29.09it/s]\n",
      "Dev: time: 0.28s, speed: 404.35st/s; acc: 0.9554, p: 0.7830, r: 0.7905, f: 0.7867 \n",
      "INFO:test:Dev: time: 0.28s, speed: 404.35st/s; acc: 0.9554, p: 0.7830, r: 0.7905, f: 0.7867 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1393  All token =  1458  acc =  0.9554183813443072\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.96it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9524_p0.7078_r0.6792_f0.6932.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9524_p0.7078_r0.6792_f0.6932.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9524_p0.7078_r0.6792_f0.6932.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9524_p0.7078_r0.6792_f0.6932.pth\n",
      "Test: time: 0.58s, speed: 323.93st/s; acc: 0.9524, p: 0.7078, r: 0.6792, f: 0.6932 \n",
      "INFO:test:Test: time: 0.58s, speed: 323.93st/s; acc: 0.9524, p: 0.7078, r: 0.6792, f: 0.6932 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3438  All token =  3610  acc =  0.9523545706371191\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 6/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 27.15it/s]\n",
      "Dev: time: 0.30s, speed: 377.20st/s; acc: 0.9643, p: 0.8491, r: 0.8571, f: 0.8531 \n",
      "INFO:test:Dev: time: 0.30s, speed: 377.20st/s; acc: 0.9643, p: 0.8491, r: 0.8571, f: 0.8531 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1406  All token =  1458  acc =  0.9643347050754458\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 19.38it/s]\n",
      "Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9590_p0.7560_r0.7254_f0.7404.pth\n",
      "INFO:test:Save current best acc model in file:/data_8t/YATO_MODEL/acc0.9590_p0.7560_r0.7254_f0.7404.pth\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9590_p0.7560_r0.7254_f0.7404.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9590_p0.7560_r0.7254_f0.7404.pth\n",
      "Test: time: 0.63s, speed: 299.36st/s; acc: 0.9590, p: 0.7560, r: 0.7254, f: 0.7404 \n",
      "INFO:test:Test: time: 0.63s, speed: 299.36st/s; acc: 0.9590, p: 0.7560, r: 0.7254, f: 0.7404 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3462  All token =  3610  acc =  0.9590027700831025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 7/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 29.04it/s]\n",
      "Dev: time: 0.28s, speed: 403.44st/s; acc: 0.9602, p: 0.8249, r: 0.8524, f: 0.8384 \n",
      "INFO:test:Dev: time: 0.28s, speed: 403.44st/s; acc: 0.9602, p: 0.8249, r: 0.8524, f: 0.8384 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1400  All token =  1458  acc =  0.9602194787379973\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.07it/s]\n",
      "Test: time: 0.60s, speed: 310.14st/s; acc: 0.9593, p: 0.7701, r: 0.7457, f: 0.7577 \n",
      "INFO:test:Test: time: 0.60s, speed: 310.14st/s; acc: 0.9593, p: 0.7701, r: 0.7457, f: 0.7577 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3463  All token =  3610  acc =  0.9592797783933518\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 8/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:00<00:00, 30.30it/s]\n",
      "Dev: time: 0.27s, speed: 421.70st/s; acc: 0.9643, p: 0.8409, r: 0.8810, f: 0.8605 \n",
      "INFO:test:Dev: time: 0.27s, speed: 421.70st/s; acc: 0.9643, p: 0.8409, r: 0.8810, f: 0.8605 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  1406  All token =  1458  acc =  0.9643347050754458\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 20.80it/s]\n",
      "Save current best f model in file:/data_8t/YATO_MODEL/acc0.9634_p0.7758_r0.7601_f0.7679.pth\n",
      "INFO:test:Save current best f model in file:/data_8t/YATO_MODEL/acc0.9634_p0.7758_r0.7601_f0.7679.pth\n",
      "Test: time: 0.58s, speed: 320.87st/s; acc: 0.9634, p: 0.7758, r: 0.7601, f: 0.7679 \n",
      "INFO:test:Test: time: 0.58s, speed: 320.87st/s; acc: 0.9634, p: 0.7758, r: 0.7601, f: 0.7679 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Right token =  3478  All token =  3610  acc =  0.9634349030470915\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Current Learning Rate: 3e-05 \n",
      "INFO:test:Current Learning Rate: 3e-05 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 9/10\n"
     ]
    }
   ],
   "source": [
    "from yato import YATO\n",
    "model = YATO('sample_data/demo.bert.config')#configuration file\n",
    "model.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c63271",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Decode    \n",
    "Decode Configuration \n",
    "\n",
    "status=decode  \n",
    "raw_dir=The path of decode file    \n",
    "nbest=0 (NER)/1 (sentence classification)   \n",
    "decode_dir=The path of decode result file  \n",
    "load_model_dir=The path of model weights          \n",
    "sentence_classification=True/False  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1217331",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from yato import YATO\n",
    "decode_model = YATO('sample_data/decode.config')#configuration file\n",
    "result_dict = decode_model.decode()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "300f4c7e",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440e98c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "### use # to comment out the configure item\n",
    "\n",
    "### I/O ###\n",
    "train_dir=demo.train.bioes\n",
    "dev_dir=demo.dev.bioes\n",
    "test_dir=demo.test.bioes\n",
    "model_dir=test/\n",
    "\n",
    "dset_dir=bert_base_cased.dset\n",
    "\n",
    "norm_word_emb=False\n",
    "norm_char_emb=False\n",
    "number_normalized=False\n",
    "seg=True\n",
    "word_emb_dim=50\n",
    "char_emb_dim=30\n",
    "\n",
    "###NetworkConfiguration###\n",
    "use_crf=False\n",
    "use_char=False\n",
    "char_seq_feature=CNN\n",
    "use_word_seq=False\n",
    "use_word_emb=True\n",
    "word_seq_feature=LSTM\n",
    "low_level_transformer=None\n",
    "low_level_transformer_finetune=False\n",
    "high_level_transformer=bert-base-cased\n",
    "high_level_transformer_finetune=True\n",
    "\n",
    "###TrainingSetting###\n",
    "status=train\n",
    "optimizer=AdamW\n",
    "iteration=10\n",
    "batch_size=16\n",
    "ave_batch_loss=False\n",
    "\n",
    "###Hyperparameters###\n",
    "cnn_layer=4\n",
    "char_hidden_dim=50\n",
    "hidden_dim=768\n",
    "dropout=0.3\n",
    "lstm_layer=2\n",
    "bilstm=True\n",
    "learning_rate=3e-5\n",
    "gpu=True\n",
    "device=cuda:0\n",
    "scheduler=get_cosine_schedule_with_warmup\n",
    "warmup_step_rate=0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6687cce",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "status=decode\n",
    "raw_dir=demo.test.bioes\n",
    "nbest=0\n",
    "decode_dir=test.decode\n",
    "load_model_dir=test/acc: 0.9777, p: 0.8973, r: 0.8973, f: 0.8973.pth      \n",
    "sentence_classification=False"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}