removing punctuations from text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Everything:\n",
    "9) **Removing punctuations from a text**\n",
    "<br>Also, getting a text file from the Internet using *request*\n",
    "\n",
    "# ورداشتن نقطه‌گون‌ها از یک مَتیان\n",
    "همچنین، نمونه ای از بارگیری یک پرونده از روی اندرتور\n",
    "<br>By Hamed Shah-Hosseini<br>\n",
    "Explanation in English:<br>\n",
    "https://www.pinterest.com/HamedShahHosseini/programming-languages/python\n",
    "<br> Explanation in Persian: https://www.instagram.com/words.persian\n",
    "<br>The code is at: https://github.com/ostad-ai/Python-Everything"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "نمایش ماریک‌های نقطه‌گون :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "# showing the punctuations\n",
    "print(string.punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "شناساندن دو کارکنش ساده، برای ورداشتن نقطه‌گون‌ها از متیان :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simple function to remove punctuation marks\n",
    "def remove_punctuations(text,ps=string.punctuation):\n",
    "    return ''.join([letter for letter in text\\\n",
    "                if letter not in ps])\n",
    "\n",
    "# simple function to include only alphanumeric\n",
    "def only_alphanumeric(text):\n",
    "    return ''.join([letter for letter in text\\\n",
    "                if letter.isalnum() or letter.isspace()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "نمونه‌ای از کاریرد کارکنش بالا، در ورداشتن نقطه‌گون‌ها :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---Text with punctuations:\n",
      " Hello world! Remove punctuation marks from this text; will you?\n",
      "By the way, Good luck.\n",
      "---------\n",
      "---Text without punctuations:\n",
      " Hello world Remove punctuation marks from this text will you\n",
      "By the way Good luck\n"
     ]
    }
   ],
   "source": [
    "#removing punctuations from a text with function remove_punctuations\n",
    "text='Hello world! Remove punctuation marks from this text;'\n",
    "text+=' will you?\\nBy the way, Good luck.'\n",
    "print('---Text with punctuations:\\n',text)\n",
    "print('---------')\n",
    "print('---Text without punctuations:\\n',remove_punctuations(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---Text with punctuations:\n",
      " Hello world! Remove punctuation marks from this text; will you?\n",
      "By the way, Good luck.\n",
      "---------\n",
      "---Text without punctuations:\n",
      " Hello world Remove punctuation marks from this text will you\n",
      "By the way Good luck\n"
     ]
    }
   ],
   "source": [
    "#removing punctuations from a text with function only_alphanumeric\n",
    "text='Hello world! Remove punctuation marks from this text;'\n",
    "text+=' will you?\\nBy the way, Good luck.'\n",
    "print('---Text with punctuations:\\n',text)\n",
    "print('---------')\n",
    "print('---Text without punctuations:\\n',only_alphanumeric(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Let's download a text file from the Internet <br>\n",
    "بیاییم از اندرتور، یک پرونده متیان را بارگیری کنیم و روی آن، ورداشتن نقطه‌گون‌ها را انجام دهیم :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file is downloaded\n"
     ]
    }
   ],
   "source": [
    "#gettign a text file from Internet\n",
    "from urllib import request\n",
    "file_url='https://raw.githubusercontent.com/ostad-ai/'+\\\n",
    "'Machine-Learning/main/alice_in_wonderland.txt'\n",
    "with request.urlopen(file_url) as file:\n",
    "    alice=file.read().decode('utf-8')\n",
    "    print('file is downloaded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---Text with punctuations:\n",
      " l as this, I shall\n",
      "think nothing of tumbling down stairs! How brave they’ll all think me\n",
      "at home! Why, I wouldn’t say anything about it, even if I fell off the\n",
      "top of the house!” (Which was very likely\n",
      "---------\n",
      "---Text without punctuations:\n",
      " l as this I shall\n",
      "think nothing of tumbling down stairs How brave they’ll all think me\n",
      "at home Why I wouldn’t say anything about it even if I fell off the\n",
      "top of the house” Which was very likely\n"
     ]
    }
   ],
   "source": [
    "text2=alice[3000:3201]\n",
    "print('---Text with punctuations:\\n',text2)\n",
    "print('---------')\n",
    "print('---Text without punctuations:\\n',remove_punctuations(text2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is seen that in above punctuation removal, something is wrong. The four below punctuations have not been removed\n",
    "<br>در ورداشتن نقطه‌گون‌ها در بالا، دیده میشود که چهار نویسه زیر، همچنان برجا مانده‌اند"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Left single quotation:  ‘\n",
      "Right single quotation:  ’\n",
      "Left double quotation:  “\n",
      "Right double quotation:  ”\n"
     ]
    }
   ],
   "source": [
    "left_single_quotation='\\U00002018'\n",
    "right_single_quotation='\\U00002019'\n",
    "left_double_quotation='\\U0000201C'\n",
    "right_double_quotation='\\U0000201D'\n",
    "print('Left single quotation: ',left_single_quotation)\n",
    "print('Right single quotation: ',right_single_quotation)\n",
    "print('Left double quotation: ',left_double_quotation)\n",
    "print('Right double quotation: ',right_double_quotation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Let's also remove the four quotations, mentioned above <br>\n",
    "اکنون چهار نویسه بالا را نیز، در رشته نقطه‌گون‌ها میگذاریم، تا از متیان داده شده، ورداشته شوند"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---Text with punctuations:\n",
      " l as this, I shall\n",
      "think nothing of tumbling down stairs! How brave they’ll all think me\n",
      "at home! Why, I wouldn’t say anything about it, even if I fell off the\n",
      "top of the house!” (Which was very likely\n",
      "---------\n",
      "---Text without punctuations:\n",
      " l as this I shall\n",
      "think nothing of tumbling down stairs How brave theyll all think me\n",
      "at home Why I wouldnt say anything about it even if I fell off the\n",
      "top of the house Which was very likely\n"
     ]
    }
   ],
   "source": [
    "#this time, we include the four quotation marks, mentioned above\n",
    "text2=alice[3000:3201]\n",
    "print('---Text with punctuations:\\n',text2)\n",
    "print('---------')\n",
    "ps=string.punctuation+'\\U00002018'+'\\U00002019'+'\\U0000201C'+'\\U0000201D'\n",
    "print('---Text without punctuations:\\n',remove_punctuations(text2,ps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "We may use function *only_alphanumeric* for the above text <br>\n",
    "میتوانیم کارکنش دیگر را برای متیان بالا به کار ببریم، و ببینیم به چه دستاوردی میرسد :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---Text with punctuations:\n",
      " l as this, I shall\n",
      "think nothing of tumbling down stairs! How brave they’ll all think me\n",
      "at home! Why, I wouldn’t say anything about it, even if I fell off the\n",
      "top of the house!” (Which was very likely\n",
      "---------\n",
      "---Text without punctuations:\n",
      " l as this I shall\n",
      "think nothing of tumbling down stairs How brave theyll all think me\n",
      "at home Why I wouldnt say anything about it even if I fell off the\n",
      "top of the house Which was very likely\n"
     ]
    }
   ],
   "source": [
    "# keeping only alphanumeric characters for alice text\n",
    "text2=alice[3000:3201]\n",
    "print('---Text with punctuations:\\n',text2)\n",
    "print('---------')\n",
    "print('---Text without punctuations:\\n',only_alphanumeric(text2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}