kadi_demo.ipynb

{
 "metadata": {
  "name": "kadi_demo"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Kadi\n",
      "====\n",
      "\n",
      "What is Kadi?\n",
      "\n",
      "- **Database** of Chandra events\n",
      "\n",
      "  - Events in telemetry such as maneuvers, NPM dwells, obsids, mech movements,\n",
      "    momentum dumps, orbit events, etc.\n",
      "  - CAPs, DSN passes, dark cals, SCS107, safe modes, bright star hold, etc\n",
      "  - Chandra major events since launch\n",
      "  - Every load command run on-board since 2002, with a link to source load products\n",
      "  - The database is contained in easily transportable sqlite3 or HDF5 files.\n",
      "\n",
      "- **Python API** for accessing events for analysis and using with the Ska engineering archive\n",
      "- **Python tools** to maintain the Kadi database on the HEAD and Greta networks \n",
      "- **Web site** for browsing events on the icxc site or by a localhost server on GRETA.  This\n",
      "  will use the [Django web framework](https://www.djangoproject.com/) to provide query and administrative capabilities.\n",
      "- **RESTful web service** API on the icxc site."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Getting events in Python\n",
      "------------------------"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from kadi import events"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import astropy.table.pprint\n",
      "astropy.table.pprint.MAX_WIDTH.set(300)\n",
      "astropy.table.pprint.MAX_LINES.set(30)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "help(events)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Find and filter\n",
      "----------------\n",
      "Each of the values in `DATA` is an `EventQuery` object that lets you access the events of that type.\n",
      "There are two object methods to do that:\n",
      "\n",
      "- `find(start, stop, **filter_kwargs)`: get a structured array for events between `start` and `stop` matching `filter_kwargs`\n",
      "- `filter(start, stop, **filter_kwargs)`: get a [Django QuerySet](https://docs.djangoproject.com/en/1.4/topics/db/queries/) for events between `start` and `stop` matching `filter_kwargs`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Find"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tsc_moves = events.tsc_moves.filter('2012:001', '2012:004').table\n",
      "print tsc_moves"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tsc_moves = events.tsc_moves.all().table  # with no args you get the full mission (from 2000)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(tsc_moves)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "impska"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out = plot_cxctime(tsc_moves['tstart'], tsc_moves['max_pwm'], '.')\n",
      "ylim(-0.5, 12)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvrs = events.manvrs.filter('2012:001').table\n",
      "manvrs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out = plot_cxctime(manvrs['tstart'], manvrs['angle'], '.')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out = hist(manvrs['angle'], bins=50)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "orbits = events.orbits.all().table\n",
      "out = plot_cxctime(orbits['tstart'], orbits['dt_start_radzone'], '.r')\n",
      "out = plot_cxctime(orbits['tstart'], orbits['dt_stop_radzone'], '.b')\n",
      "grid()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "maj_evts = events.major_events.filter('2011:180', '2011:190').table\n",
      "maj_evts['start', 'date', 'descr', 'source']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print events.major_events.filter(descr__icontains='safemode')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print events.major_events.filter(descr__icontains='safe', source__exact='FOT')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Filter\n",
      "\n",
      "`filter()` brings you the full power of the Django **Object Relational Mapper**\n",
      "\n",
      "This includes the ability to chain filters, but more important the objects that are returned are not simple numpy structured arrays.  Instead they are model objects that can support capabilities like plotting and one-to-many database relationships.  This will be clear from the following examples."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scs107s = events.major_events.filter(descr__icontains='scs #107', source__exact='FOT')\n",
      "print type(scs107s)\n",
      "print type(scs107s[0])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(scs107s)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scs107s.table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "events.major_events.filter(descr__icontains='scs', source__exact='FOT').table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(events.major_events.filter(descr__icontains='scs', source__exact='FOT').filter(descr__icontains='107'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvrs = events.manvrs.filter('2011:001', '2011:030', n_dwell__exact=1, angle__gte=140)\n",
      "for manvr in manvrs:\n",
      "    print manvr"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted({manvr.template for manvr in events.manvrs.filter()})"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr = manvrs[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "help(manvr)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr.plot()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for manvrseq in manvr.manvrseq_set.all():\n",
      "    print manvrseq"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr.manvrseq_set.all().table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Commands\n",
      "--------\n",
      "\n",
      "All the commands that have been run as stored commands via SCS slots 128 to 133 are stored in a single HDF5 file.  These are accessible via the `kadi.cmds` module.  (*To be changed to kadi.commands*).\n",
      "\n",
      "Technically speaking, \"all commands\" refers to every command that is part of a load segment that is on iFOT and was actually uplinked and run.  This is part of the Ska commanded states database.  There is a certain component of human caretaking to ensure that load stop times are correct, so small errors are possible (and likely) near times of anomalous load stops.\n",
      "\n",
      "One final limitation is that the star catalog parameters are not stored.  To do so would increase the file size by a factor of 10."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from kadi import cmds as commands"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Find and filter\n",
      "\n",
      "The commands interface also features `find` and `filter` functions.  They share the same idea as the corresponding `events` versions, but the details are different."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**find**: `find(start, stop, **filter_kwargs)` returns a numpy array with `idx`, `date`, `type`, `tlmsid`, `scs`, `step`, and `timeline_id` fields"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "commands.filter('2012:001:00:00:00', '2012:001:01:00:00').table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pickle\n",
      "cmd_params = pickle.load(open('cmds.pkl'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(cmd_params)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cmd_params.items()[-1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rev_params = {v: k for k, v in cmd_params.items()}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rev_params[34]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "commands.filter('2012:001', '2012:005', tlmsid='WSPOW00000').table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "commands.filter('2012:001', '2012:002', tlmsid='AOUPTARQ').table"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Filter**: `filter(start, stop, **filter_kwargs)` returns a `CmdList` object with **all** the parameters"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cmds = commands.filter('2012:001', '2012:002', tlmsid='AOUPTARQ')\n",
      "print type(cmds)\n",
      "print cmds"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Intervals\n",
      "----------\n",
      "\n",
      "The `QueryEvent` class provides a way to define complex bad time filters to the Ska engineering archive."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr_intervals = events.manvrs.intervals('2012:001', '2012:002')\n",
      "manvr_intervals"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "manvr_intervals = events.manvrs.intervals('2012:001', '2012:002')\n",
      "manvr_intervals"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dwell_intervals = events.dwells.intervals('2012:001', '2012:002')\n",
      "dwell_intervals"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "(events.manvrs & events.tsc_moves).intervals('2012:001', '2012:002')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dwells = events.dwells\n",
      "dwells.interval_pad = -100\n",
      "tsc_moves = events.tsc_moves\n",
      "tsc_moves.interval_pad = 300\n",
      "dumps = events.dumps\n",
      "dumps.interval_pad = (10, 500)\n",
      "good_times = dwells & ~(tsc_moves | dumps)\n",
      "good_times.intervals('2012:001', '2012:002')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dat = fetch.Msid('aoattqt1', '2012:001', '2012:002')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dat.select_intervals(good_times)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out = plot_cxctime(dat.times, dat.vals, '.')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rad_zones = events.rad_zones\n",
      "rad_zone_dwells = rad_zones & dwells\n",
      "rad_zone_dwells.intervals('2012:001', '2012:007')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "altitude = fetch.Msid('Dist_SatEarth', '2012:001', '2012:007')\n",
      "altitude.select_intervals(rad_zones)\n",
      "plot_cxctime(altitude.times, altitude.vals, '.')\n",
      "grid()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}