update the tox

Author: hawkowl

README.rst

@@ -10,8 +10,8 @@ This README describes the steps involved in installing and running Optima. **Fol
 
 0. Download and install the GitHub app (http://desktop.github.com) on Windows or Mac, or `sudo apt-get install git` on Linux.
 
-0. Go to the Optima GitHub page and click on the small button next to "Download ZIP":  
-![](http://optimamodel.com/figs/optima-github-button.png)  
+0. Go to the Optima GitHub page and click on the small button next to "Download ZIP":
+![](http://optimamodel.com/figs/optima-github-button.png)
 (or use `git clone` if on Linux)
 
 0. Finally, set up the Python path:
@@ -32,11 +32,11 @@ This README describes the steps involved in installing and running Optima. **Fol
 
 ## 2.1 Quick start installation
 
-To install, run `python setup.py develop` in the root repository directory. This will add Optima to the system path. Optima can then be used via Python. 
+To install, run `python setup.py develop` in the root repository directory. This will add Optima to the system path. Optima can then be used via Python.
 
 To uninstall, run `python setup.py develop --uninstall`.
 
-Note: do **not** use `python setup.py install`, as this will copy the source code into your system Python directory, and you won't be able to modify or update it easily. 
+Note: do **not** use `python setup.py install`, as this will copy the source code into your system Python directory, and you won't be able to modify or update it easily.
 
 
 ## 2.2 Detailed instructions
@@ -64,25 +64,25 @@ The last step is to make sure that Optima is available on the Python path. There
     0. Under the “Tools” (Linux and Windows) or “python” (under Mac) menu, go to “PYTHONPATH Manager”
     0. Select the Optima folder (e.g. `C:\Users\Alice\GitHub\Optima` on Windows) and click OK.
  0. **Option 2: modify system path**
-    0. **Option 2A** (all operating systems): Go to the Optima root folder (in a terminal on Mac or Linux; in a command prompt [cmd.exe] in Windows) and run  
-    `python setup.py develop`  
+    0. **Option 2A** (all operating systems): Go to the Optima root folder (in a terminal on Mac or Linux; in a command prompt [cmd.exe] in Windows) and run
+    `python setup.py develop`
     Note: if Spyder does not use the system Python (which can happen in some cases), this will not work. In this case:
-       0. Inside a Spyder console, type  
+       0. Inside a Spyder console, type
           `import sys; sys.executable`
-       0. Replace the above command with the location of this executable, e.g.  
+       0. Replace the above command with the location of this executable, e.g.
           `/software/anaconda/bin/python setup.py develop`
-    0. **Option 2B** (Linux, Mac only): Add the Optima folder to `~/.bashrc` or `~/.bash_profile`, e.g.  
-    `export PYTHONPATH=$PYTHONPATH:/users/alice/github/optima`  
+    0. **Option 2B** (Linux, Mac only): Add the Optima folder to `~/.bashrc` or `~/.bash_profile`, e.g.
+    `export PYTHONPATH=$PYTHONPATH:/users/alice/github/optima`
     [NB: if you don't use `bash`, you are probably a hacker and don't need these instructions.]
-    0. **Option 2C** (Windows only): search for “variables” from the Start Menu; the option should be called something like “Edit environment variables for your account”. Under “user variables”, you should see “PYTHONPATH” listed. Add the folder for the Optima repository, e.g.  
-    `C:\Users\Alice\GitHub\Optima`  
-    If there are already things on the Python path, add this to the end separated by a semicolon and no space, e.g.  
+    0. **Option 2C** (Windows only): search for “variables” from the Start Menu; the option should be called something like “Edit environment variables for your account”. Under “user variables”, you should see “PYTHONPATH” listed. Add the folder for the Optima repository, e.g.
+    `C:\Users\Alice\GitHub\Optima`
+    If there are already things on the Python path, add this to the end separated by a semicolon and no space, e.g.
     `C:\Anaconda2\Library\bin;C:\Users\Alice\GitHub\Optima`
 
 ### 2.3 Verification/usage
 
-If you followed the steps correctly, you should be able to run  
-`import optima`  
+If you followed the steps correctly, you should be able to run
+`import optima`
 from a Python console (either the system console or the Spyder console)
 
 For usage examples, see the scripts in the `tests` folder. In particular, `testworkflow.py` shows a typical usage example.
@@ -101,7 +101,7 @@ On mac, install the `postgres` software with:
 
     brew install postgres
 
-On Linux, use 
+On Linux, use
 
     sudo apt-get install install postgres
 
@@ -114,7 +114,7 @@ To run the `postgres` daemon in a terminal:
 ```bash
 postgres -D /usr/local/var/postgresbrew
 ```
-    
+
 If you want to, you can run the `postgres` daemon with the Mac system daemon manager `launchctl`, or via the ruby wrapper for `lunchy`.
 
 
@@ -141,7 +141,7 @@ You will first need to install the python database migration tools:
 ```bash
 pip install sqlalchemy-migrate psycopg2
 ```
-	
+
 Then to create the optima database, use these commands *from the root Optima directory* as `migrate` needs to find the migration scripts:
 
 ```bash
@@ -225,7 +225,9 @@ _On Mac_:
     lunchy start redis
 
 
-Run the server in two separate terminals. These scripts will start Python in a `virtualenv` isolated Python environments. First in one terminal:
+Run the server in two separate terminals. These scripts will start Python in a `virtualenv` isolated Python environments.
+If you wish to use system installed packages, append `--sitepackages` and it will not reinstall things that are already installed in the Python site packages.
+First in one terminal:
 
     tox -e celery
 
@@ -270,10 +272,10 @@ This section contains random pieces of wisdom we have encountered along the way.
 ## 7.1 Workflows
 
 - Make sure you pull and push from the repository regularly so you know what everyone else is doing, and everyone else knows what you're doing. If your branch is 378 commits behind develop, you're the sucker who's going to have to merge it.
-- There is very unclear advice about how to debug Python. It's actually fairly simple: if you run Python in interactive mode (e.g. via Spyder or via `python -i`), then if a script raises an exception, enter this in the console just after the crash:  
-`import pdb; pdb.pm()`   
-You will then be in a debugger right where the program crashed. Type `?` for available commands, but it works like how you would expect. Alternatively, if you want to effectively insert a breakpoint into your program, you can do this with  
-`import pdb; pdb.set_trace()`  
+- There is very unclear advice about how to debug Python. It's actually fairly simple: if you run Python in interactive mode (e.g. via Spyder or via `python -i`), then if a script raises an exception, enter this in the console just after the crash:
+`import pdb; pdb.pm()`
+You will then be in a debugger right where the program crashed. Type `?` for available commands, but it works like how you would expect. Alternatively, if you want to effectively insert a breakpoint into your program, you can do this with
+`import pdb; pdb.set_trace()`
 No one knows what these mysterious commands do. Just use them.
 - For benchmarking/profiling, you can use `tests/benchmarkmodel.py`. It's a good idea to run this and see if your changes have slowed things down considerably. It shows how to use the line profiler; Spyder also comes with a good function-level (but not line) profiler.
 
@@ -291,7 +293,7 @@ def myfunc(args=None):
   if args is None: args = []
   print(args)
 ```
-- It's dangerous to use `type()`; safer to use `isinstance()` (unless you _really_ mean `type()`). For example,   
-`type(rand(1)[0])==float`  
-is `False` because its type is `<type 'numpy.float64'>`; use `isinstance()` instead, e.g.   `isinstance(rand(1)[0], (int, float))`  
+- It's dangerous to use `type()`; safer to use `isinstance()` (unless you _really_ mean `type()`). For example,
+`type(rand(1)[0])==float`
+is `False` because its type is `<type 'numpy.float64'>`; use `isinstance()` instead, e.g.   `isinstance(rand(1)[0], (int, float))`
  will catch anything that looks like a number, which is usually what you _really_ want.

bin/run_server.py

@@ -0,0 +1,12 @@
+import sys
+import os
+
+try:
+    import server
+except:
+    sys.path.insert(0, os.path.abspath("."))
+
+from server import _autoreload
+from server import _twisted_wsgi
+
+_autoreload.main(_twisted_wsgi.run)

server/_autoreload.py

@@ -0,0 +1,141 @@
+# Autoreloading launcher.
+# Borrowed from Peter Hunt and the CherryPy project (http://www.cherrypy.org).
+# Some taken from Ian Bicking's Paste (http://pythonpaste.org/).
+# Adjustments made by Michael Elsdoerfer ([email protected]).
+#
+# Portions copyright (c) 2004, CherryPy Team ([email protected])
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification,
+# are permitted provided that the following conditions are met:
+#
+#     * Redistributions of source code must retain the above copyright notice,
+#       this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright notice,
+#       this list of conditions and the following disclaimer in the documentation
+#       and/or other materials provided with the distribution.
+#     * Neither the name of the CherryPy Team nor the names of its contributors
+#       may be used to endorse or promote products derived from this software
+#       without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import os, sys, time
+
+try:
+    import thread
+except ImportError:
+    import dummy_thread as thread
+
+# This import does nothing, but it's necessary to avoid some race conditions
+# in the threading module. See http://code.djangoproject.com/ticket/2330 .
+try:
+    import threading
+except ImportError:
+    pass
+
+
+RUN_RELOADER = True
+
+_mtimes = {}
+_win = (sys.platform == "win32")
+
+def code_changed():
+    global _mtimes, _win
+    for filename in filter(lambda v: v, map(lambda m: getattr(m, "__file__", None), sys.modules.values())):
+        if filename.endswith(".pyc") or filename.endswith(".pyo"):
+            filename = filename[:-1]
+        if not os.path.exists(filename):
+            continue # File might be in an egg, so it can't be reloaded.
+        stat = os.stat(filename)
+        mtime = stat.st_mtime
+        if _win:
+            mtime -= stat.st_ctime
+        if filename not in _mtimes:
+            _mtimes[filename] = mtime
+            continue
+        if mtime != _mtimes[filename]:
+            _mtimes = {}
+            return True
+    return False
+
+def reloader_thread(softexit=False):
+    """If ``soft_exit`` is True, we use sys.exit(); otherwise ``os_exit``
+    will be used to end the process.
+    """
+    while RUN_RELOADER:
+        if code_changed():
+            # force reload
+            if softexit:
+                sys.exit(3)
+            else:
+                os._exit(3)
+        time.sleep(1)
+
+def restart_with_reloader():
+    while True:
+        args = [sys.executable] + sys.argv
+        if sys.platform == "win32":
+            args = ['"%s"' % arg for arg in args]
+        new_environ = os.environ.copy()
+        new_environ["RUN_MAIN"] = 'true'
+        exit_code = os.spawnve(os.P_WAIT, sys.executable, args, new_environ)
+        if exit_code != 3:
+            return exit_code
+
+def python_reloader(main_func, args, kwargs, check_in_thread=True):
+    """
+    If ``check_in_thread`` is False, ``main_func`` will be run in a separate
+    thread, and the code checker in the main thread. This was the original
+    behavior of this module: I (Michael Elsdoerfer) changed the default
+    to be the reverse: Code checker in thread, main func in main thread.
+    This was necessary to make the thing work with Twisted
+    (http://twistedmatrix.com/trac/ticket/4072).
+    """
+    if os.environ.get("RUN_MAIN") == "true":
+        if check_in_thread:
+            thread.start_new_thread(reloader_thread, (), {'softexit': False})
+        else:
+            thread.start_new_thread(main_func, args, kwargs)
+
+        try:
+            if not check_in_thread:
+                reloader_thread(softexit=True)
+            else:
+                main_func(*args, **kwargs)
+        except KeyboardInterrupt:
+            pass
+    else:
+        try:
+            sys.exit(restart_with_reloader())
+        except KeyboardInterrupt:
+            pass
+
+def jython_reloader(main_func, args, kwargs):
+    from _systemrestart import SystemRestart
+    thread.start_new_thread(main_func, args)
+    while True:
+        if code_changed():
+            raise SystemRestart
+        time.sleep(1)
+
+
+def main(main_func, args=None, kwargs=None, **more_options):
+    if args is None:
+        args = ()
+    if kwargs is None:
+        kwargs = {}
+    if sys.platform.startswith('java'):
+        reloader = jython_reloader
+    else:
+        reloader = python_reloader
+    reloader(main_func, args, kwargs, **more_options)

server/twisted_server.py server/_twisted_wsgi.py

@@ -8,13 +8,15 @@
 from twisted.web.server import Site
 from twisted.web.static import File
 from twisted.web.wsgi import WSGIResource
+from twisted.python.threadpool import ThreadPool
 
 globalLogBeginner.beginLoggingTo([
     FileLogObserver(sys.stdout, lambda _: formatEvent(_) + "\n")])
 
 import api
 
-wsgi_app = WSGIResource(reactor, reactor.getThreadPool(), api.app)
+threadpool = ThreadPool(maxthreads=30)
+wsgi_app = WSGIResource(reactor, threadpool, api.app)
 
 class OptimaResource(Resource):
     isLeaf = True
@@ -39,7 +41,15 @@ def render(self, request):
 except IndexError:
     port = "8080"
 
-endpoint = serverFromString(reactor, "tcp:port=" + port)
-endpoint.listen(site)
 
-reactor.run()
+def run():
+    """
+    Run the server.
+    """
+    endpoint = serverFromString(reactor, "tcp:port=" + port)
+    endpoint.listen(site)
+
+    reactor.run()
+
+if __name__ == "__main__":
+    run()

tox.ini

@@ -1,14 +1,16 @@
 [tox]
-envlist = runserver, celery
+envlist = runserver, runserver-prod, celery, celery-prod
 
-[testenv:runserver]
+[testenv]
 changedir = {toxinidir}
+
 commands =
     pip install -r server/requirements.txt
-    python -m server.twisted_server {posargs}
+    {runserver}: python bin/run_server.py {posargs}
+    {celery}: celery -A server.webapp.tasks.celery_instance worker -l info
+
+[testenv:runserver]
+skip_install = True
 
 [testenv:celery]
-changedir = {toxinidir}
-commands =
-    pip install -r server/requirements.txt
-    celery -A server.webapp.tasks.celery_instance worker -l info
+skip_install = True