tutorials/notebooks/tbr-files.v0x04.run.length.encoding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Refinery Files 0x04: Run Length Encoding\n",
    "\n",
    "In this tutorial, we will be looking at a malicious loader sample that uses [run-length encoding][RLE] (RLE) to compress its payload. Unfortunately, I am not aware of any specification for RLE algorithms, so it would be difficult to write an RLE unit that covers every possible implementation. The goal of this tutorial is to highlight how easy it is to write a custom refinery unit for cases like this.\n",
    "\n",
    "[RLE]: https://en.wikipedia.org/wiki/Run-length_encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tutorials import boilerplate\n",
    "boilerplate.store_sample('376983700af2d2109c58d8c8c99ecef8634a1276a8eead2bb509c6627ffde456', 'a.bin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00.256 MB 376983700af2d2109c58d8c8c99ecef8634a1276a8eead2bb509c6627ffde456 a.bin\n"
     ]
    }
   ],
   "source": [
    "%ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the binary in a hex editor, you quickly spot a base64-encoded buffer starting at offset `0x158F0`, corresponding to the virtual address `0x1800168F0`. We know how to extract that using our old friend [carve]:\n",
    "\n",
    "[carve]: https://binref.github.io/#refinery.carve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------------------------------------\n",
      "00.111 MB; 97.09% entropy; data\n",
      "------------------------------------------------------------------------------------------------------------------------\n",
      "00000: 83 3D 31 FC 66 7A 73 6B 9C 75 6D 7A 6B 9A 75 73 90 CE 65 65 87 2B 59 61 99 8B 6B 6B  .=1.fzsk.umzk.us..ee.+Ya..kk\n",
      "0001C: 8F 63 6B D1 76 70 DF 65 AB 58 C8 6A 2F BC 4F 2E 01 0C 04 53 1F 04 0D 02 0A 0A 17 41  .ck.vp.e.X.j/.O....S.......A\n",
      "00038: 05 0A 6A 05 86 02 00 4B 1A 15 4B 1E 13 17 50 02 0D 51 2A 35 3A 45 1A 1C 0B 13 4C 67  ..j....K..K...P..Q*5:E....Lg\n",
      "00054: 75 95 70 45 61 6B D0 A1 24 14 0F E5 51 67 43 E2 4F 6E 58 E5 4A 66 46 AF 28 73 5E FE  u.pEak..$...QgC.OnX.JfF.(s^.\n",
      "00070: 46 61 4A EB 51 7D 52 E5 4F 7C 40 52 34 73 5D F6 51 67 43 55 3E 6E 59 E4 4A 66 46 43  FaJ.Q}R.O|@R4s].QgCU>nY.JfFC\n",
      "0008C: 31 8D 5F FC 46 61 4A 5C 20 7E 53 EE 4F 7C 40 39 05 0E 1C E5 51 67 43 74 66 87 20 2E  1._.FaJ\\.~S.O|@9....QgCtf...\n",
      "000A8: 61 71 96 1E EF 63 77 4C 2A F8 03 6D 78 9E 8A 61 44 4B 63 69 62 76 74 E5 79 72 6B 93  aq...cwL*..mx..aDKcibvt.yrk.\n",
      "000C4: 78 7F 70 95 47 11 6D 7A 96 75 72 73 91 F6 63 61 78 94 6A 62 66 94 6A 69 6C 92 72 6C  x.p.G.mz.urs..cax.jbf.jil.rl\n",
      "000E0: 78 8F 6D 64 66 87 80 6A 61 71 91 7E 6F 65 8B 71 6F 16 63 67 78 94 6A 67 66 94 78 63  x.mdf..jaq.~oe.qo.cgx.jgf.xc\n",
      "000FC: 6C 92 64 6D 78 8F 7B 66 66 86 60 68 63 8B 1E D0 68 65 4F 72 6D 76 99 CD D3 6A 7A 49  l.dmx.{ff.`hc...heOrmv...jzI\n",
      "------------------------------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "%emit a.bin | carve -sd b64 | peek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, that looks like it is encrypted and/or compressed, so we will have to do a bit of reversing. Searching for cross-references to the string address, we quickly identify that the function at `0x180003470` performs base64-decoding of the payload. Looking for cross-references to that function, we note that it called from all the various exports of the sample, and the call is always followed by a call to the function at `0x180001000`, which turns out to be the routine that decrypts and decompresses the payload.\n",
    "\n",
    "This function first performs an XOR-decryption with the following 32-byte ASCII key stored at `0x180016750`:\n",
    "```\n",
    "xpklfypkcqnziewsovbexkzafkhklmtk\n",
    "```\n",
    "That is straightforward enough, let's have a look at what the decrypted payload looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------------------------------------\n",
      "00.111 MB; 85.83% entropy; data\n",
      "------------------------------------------------------------------------------------------------------------------------\n",
      "00000: FB 4D 5A 90 00 03 03 00 FF 04 03 00 02 FF 02 00 FF B8 07 00 FF 40 23 00 FF E0 03 00  .MZ..................@#.....\n",
      "0001C: E3 0E 1F BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68 69 73 20 70 72 6F 67 72 61 6D 20  .........!..L.!This.program.\n",
      "00038: 63 61 02 6E EA 6F 74 20 62 65 20 72 75 6E 20 69 6E 20 44 4F 53 20 6D 6F 64 65 2E 02  ca.n.ot.be.run.in.DOS.mode..\n",
      "00054: 0D FE 0A 24 07 00 B8 CA 48 79 7B 8E 29 17 28 8E 29 17 28 8E 29 17 28 D5 41 16 29 8D  ...$....Hy{.).(.).(.).(.A.).\n",
      "00070: 29 17 28 8E 29 16 28 84 29 17 28 39 58 1E 29 9D 29 17 28 39 58 17 29 8F 29 17 28 39  ).(.).(.).(9X.).).(9X.).).(9\n",
      "0008C: 58 E8 28 8F 29 17 28 39 58 15 29 8F 29 17 28 52 69 63 68 8E 29 17 28 18 00 FE 50 45  X.(.).(9X.).).(Rich.).(...PE\n",
      "000A8: 02 00 F8 64 86 06 00 3F 45 8E 61 08 00 F5 F0 00 22 20 0B 02 0E 1B 00 8E 01 02 00 FF  ...d...?E.a.....\"...........\n",
      "000C4: 1E 06 00 FE 24 60 03 00 FF 10 05 00 FE 80 01 04 00 FF 10 03 00 FF 02 02 00 FF 06 07  ....$`......................\n",
      "000E0: 00 FF 06 08 00 FE F0 01 02 00 FF 04 06 00 FC 02 00 60 01 02 00 FF 10 06 00 FF 10 08  .................`..........\n",
      "000FC: 00 FF 10 06 00 FF 10 0A 00 FF 10 03 00 FA 70 AA 01 00 38 01 02 00 FB A8 AB 01 00 28  ..............p...8........(\n",
      "------------------------------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "%emit a.bin | carve -sd b64 | xor xpklfypkcqnziewsovbexkzafkhklmtk | peek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's almost a bingo, but note that the file magic still spells out \"data\", rather than identifying this buffer as a PE file. Also, there is a suspiciously small amount of zero bytes visible here. On the other hand, the entropy is only slightly higher than what you would expect from a code-heavy PE file (most unpacked malware falls into this category and has around 82% entropy, because that is the entropy of x86 opcodes). All of this, and a few minutes spent reverse engineering later are enough to convince us that the payload is compressed with an RLE algorithm. It works as follows:\n",
    "\n",
    "1. Read a control byte `C` from the input buffer.\n",
    "2. If the most significant bit of this control byte is set, transfer `(0x100 - C)` bytes from the input to the output buffer.\n",
    "3. Otherwise, read a repeated byte `B` from the input buffer and write `C` copies of `B` to the output buffer.\n",
    "\n",
    "It is implemented by the following unit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!binary-refinery\n",
    "from itertools import islice, repeat\n",
    "from refinery import Unit\n",
    "\n",
    "class rle(Unit):\n",
    "    def process(self, data: bytearray):\n",
    "        input, output = iter(data), bytearray()\n",
    "        for length in input:\n",
    "            if length & 0b10000000:\n",
    "                chunk = islice(input, 0x100 - length)\n",
    "            else:\n",
    "                chunk = repeat(next(input), length)\n",
    "            output[len(output):] = chunk\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the shebang in the first line:\n",
    "```py\n",
    "#!binary-refinery\n",
    "```\n",
    "For this to work, I maintain an alias called `binary-refinery` that points to the `python` interpreter of the virtual environment where binary refinery is installed. This way, when I want to write a custom refinery unit, all I have to do is to add this shebang to a custom unit I am writing. Under Windows, I maintain a directory that is added to the system `PATH`, where I can place batch scripts that act as aliases. The batch-file alias for binary refinery looks as follows:\n",
    "```bat\n",
    "@x:\\projects\\refinery\\venv\\Scripts\\python.exe %*\n",
    "```\n",
    "The rest of the script is very straightforward. All you have to do is import the `Unit` class from `refinery` and create a class that inherits from it. That class has to define a method called `process` which gets a single argument representing a chunk of data to be processed. This method can `yield` multiple byte strings as outputs or return a single byte string if it returns a single output, like in this case.\n",
    "\n",
    "And then, we simply place the above code in a file called `rle.py`, make it executable, and voila:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------------------------------------\n",
      "    crc32 = e07276b9\n",
      "  entropy = 81.80%\n",
      "    magic = PE32+ executable (DLL) (GUI) x86-64, for MS Windows\n",
      "   sha256 = e0839679e902ddbbb3f0b4d671c64549ccdd2e5f1e8a4415ad973f0a389e874b\n",
      "     size = 00.110 MB\n",
      "------------------------------------------------------------------------------------------------------------------------\n",
      "00000: 4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00 B8 00 00 00 00 00 00 00 40 00 00 00  MZ......................@...\n",
      "0001C: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ............................\n",
      "00038: 00 00 00 00 E0 00 00 00 0E 1F BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68 69 73 20 70  ................!..L.!This.p\n",
      "00054: 72 6F 67 72 61 6D 20 63 61 6E 6E 6F 74 20 62 65 20 72 75 6E 20 69 6E 20 44 4F 53 20  rogram.cannot.be.run.in.DOS.\n",
      "00070: 6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00 CA 48 79 7B 8E 29 17 28 8E 29 17 28  mode....$........Hy{.).(.).(\n",
      "------------------------------------------------------------------------------------------------------------------------\n",
      "Header.Machine    : AMD64\n",
      "Header.Subsystem  : Windows GUI\n",
      "Header.MinimumOS  : Windows Vista\n",
      "Header.ExportName : l_dll_rndll_il_64_p2_25_v141_12_11_21_logs_no.dll\n",
      "Header.RICH[0x0]  : [0101685b] 03 IMPORT Visual Studio 2017 14.15\n",
      "Header.RICH[0x1]  : [00010000] 10 IMPORT Visual Studio\n",
      "Header.RICH[0x2]  : [010971b7] 19 STDLIB Visual Studio 2019 16.2.0 preview 4\n",
      "Header.RICH[0x3]  : [010071b7] 01 EXPORT Visual Studio 2019 16.2.0 preview 4\n",
      "Header.RICH[0x4]  : [00ff71b7] 01 CVTRES Visual Studio 2019 16.2.0 preview 4\n",
      "Header.RICH[0x5]  : [010271b7] 01 LINKER Visual Studio 2019 16.2.0 preview 4\n",
      "Header.Type       : DLL\n",
      "Header.ImageBase  : 0x0000000180000000\n",
      "Header.ImageSize  : 110080\n",
      "Header.Bits       : 64\n",
      "Header.EntryPoint : 0x0000000180006024\n",
      "TimeStamp.Linker  : 2021-11-12 10:43:11\n",
      "TimeStamp.DbgDir  : 2021-11-12 10:43:11\n",
      "TimeStamp.Export  : 2106-02-07 06:28:15\n"
     ]
    }
   ],
   "source": [
    "%emit a.bin | carve -sd b64 | xor xpklfypkcqnziewsovbexkzafkhklmtk | ./rle.py | peek -mml5 | pemeta -t"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "4bb4b02bf57a2c25456a741474d02d2de926aec3c451f22b312ec34f66909bb4"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit (system)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}