Controlling neovim from jupyter notebook: "another loop is running"

[beyarkay]

I've got a jupyter notebook, from which I want to control an instance of neovim. Following the instructions in the pynvim repo, I can setup neovim to listen like so:

NVIM_LISTEN_ADDRESS=/tmp/nvim nvim

and then I can control that instance of neovim from a regular (non-notebook) python session like so:

from pynvim import attach
nvim = attach('socket', path='/tmp/nvim')
nvim.command('vsplit')

The above all works fine from a regular python session (python3 or ipython) but it doesn't work from a jupyter notebook. Trying to run the following cell:

from pynvim import attach
nvim = attach('socket', path='/tmp/nvim')

Results in the following error:

 RuntimeError: Cannot run the event loop while another loop is running 

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [9], in <cell line: 3>()
      1 # %pip install pynvim
      2 from pynvim import attach
----> 3 nvim = attach('socket', path='/tmp/nvim')

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/__init__.py:111, in attach(session_type, address, port, path, argv, decode)
     80 def attach(session_type, address=None, port=None,
     81            path=None, argv=None, decode=None):
     82     """Provide a nicer interface to create python api sessions.
     83 
     84     Previous machinery to create python api sessions is still there. This only
   (...)
    108 
    109     """
    110     session = (tcp_session(address, port) if session_type == 'tcp' else
--> 111                socket_session(path) if session_type == 'socket' else
    112                stdio_session() if session_type == 'stdio' else
    113                child_session(argv) if session_type == 'child' else
    114                None)
    116     if not session:
    117         raise Exception('Unknown session type "%s"' % session_type)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:35, in socket_session(path)
     33 def socket_session(path):
     34     """Create a msgpack-rpc session from a unix domain socket."""
---> 35     return session('socket', path)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:19, in session(transport_type, *args, **kwargs)
     18 def session(transport_type='stdio', *args, **kwargs):
---> 19     loop = EventLoop(transport_type, *args, **kwargs)
     20     msgpack_stream = MsgpackStream(loop)
     21     async_session = AsyncSession(msgpack_stream)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:93, in BaseEventLoop.__init__(self, transport_type, *args)
     91 except Exception as e:
     92     self.close()
---> 93     raise e
     94 self._start_reading()

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:90, in BaseEventLoop.__init__(self, transport_type, *args)
     88 self._init()
     89 try:
---> 90     getattr(self, '_connect_{}'.format(transport_type))(*args)
     91 except Exception as e:
     92     self.close()

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/asyncio.py:96, in AsyncioEventLoop._connect_socket(self, path)
     94 else:
     95     coroutine = self._loop.create_unix_connection(self._fact, path)
---> 96 self._loop.run_until_complete(coroutine)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:622, in BaseEventLoop.run_until_complete(self, future)
    611 """Run until the Future is done.
    612 
    613 If the argument is a coroutine, it is wrapped in a Task.
   (...)
    619 Return the Future's result, or raise its exception.
    620 """
    621 self._check_closed()
--> 622 self._check_running()
    624 new_task = not futures.isfuture(future)
    625 future = tasks.ensure_future(future, loop=self)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:584, in BaseEventLoop._check_running(self)
    582     raise RuntimeError('This event loop is already running')
    583 if events._get_running_loop() is not None:
--> 584     raise RuntimeError(
    585         'Cannot run the event loop while another loop is running')

RuntimeError: Cannot run the event loop while another loop is running

Which looks like both jupyter and pynvim are trying to run async loops at the same time. I'm not sure how to get around the issue though.

How can I control neovim from a python jupyter notebook?

I'm running on a Mac, Python 3.11.3, IPython version 8.4.0, ipykernel version 6.15.1, pynvim version 0.4.3

[beyarkay]

GH doesn't support uploading ipynb files, so here's the raw text of a minimal Jupyter notebook to reproduce the issue:

Expand me for the notebook

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5636335d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-15T10:31:53.742817Z",
     "start_time": "2023-09-15T10:31:42.902843Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pynvim in /Users/brk/Library/Python/3.10/lib/python/site-packages (0.4.3)\n",
      "Requirement already satisfied: msgpack>=0.5.0 in /Users/brk/Library/Python/3.10/lib/python/site-packages (from pynvim) (1.0.5)\n",
      "Requirement already satisfied: greenlet in /Users/brk/Library/Python/3.10/lib/python/site-packages (from pynvim) (2.0.2)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Cannot run the event loop while another loop is running",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Input \u001b[0;32mIn [5]\u001b[0m, in \u001b[0;36m<cell line: 4>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpynvim\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m attach\n\u001b[1;32m      3\u001b[0m blocker \u001b[38;5;241m=\u001b[39m \u001b[38;5;28minput\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 4\u001b[0m nvim \u001b[38;5;241m=\u001b[39m \u001b[43mattach\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msocket\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpath\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m/tmp/nvim\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m nvim\u001b[38;5;241m.\u001b[39mcommand(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mvsplit\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/__init__.py:111\u001b[0m, in \u001b[0;36mattach\u001b[0;34m(session_type, address, port, path, argv, decode)\u001b[0m\n\u001b[1;32m     80\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mattach\u001b[39m(session_type, address\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, port\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m     81\u001b[0m            path\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, argv\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, decode\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[1;32m     82\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Provide a nicer interface to create python api sessions.\u001b[39;00m\n\u001b[1;32m     83\u001b[0m \n\u001b[1;32m     84\u001b[0m \u001b[38;5;124;03m    Previous machinery to create python api sessions is still there. This only\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    108\u001b[0m \n\u001b[1;32m    109\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m    110\u001b[0m     session \u001b[38;5;241m=\u001b[39m (tcp_session(address, port) \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtcp\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[0;32m--> 111\u001b[0m                \u001b[43msocket_session\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124msocket\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m    112\u001b[0m                stdio_session() \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstdio\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m    113\u001b[0m                child_session(argv) \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mchild\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m    114\u001b[0m                \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m    116\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m session:\n\u001b[1;32m    117\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mUnknown session type \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m%\u001b[39m session_type)\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:35\u001b[0m, in \u001b[0;36msocket_session\u001b[0;34m(path)\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msocket_session\u001b[39m(path):\n\u001b[1;32m     34\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Create a msgpack-rpc session from a unix domain socket.\"\"\"\u001b[39;00m\n\u001b[0;32m---> 35\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43msession\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msocket\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpath\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:19\u001b[0m, in \u001b[0;36msession\u001b[0;34m(transport_type, *args, **kwargs)\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msession\u001b[39m(transport_type\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstdio\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m---> 19\u001b[0m     loop \u001b[38;5;241m=\u001b[39m \u001b[43mEventLoop\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransport_type\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     20\u001b[0m     msgpack_stream \u001b[38;5;241m=\u001b[39m MsgpackStream(loop)\n\u001b[1;32m     21\u001b[0m     async_session \u001b[38;5;241m=\u001b[39m AsyncSession(msgpack_stream)\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:93\u001b[0m, in \u001b[0;36mBaseEventLoop.__init__\u001b[0;34m(self, transport_type, *args)\u001b[0m\n\u001b[1;32m     91\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     92\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclose()\n\u001b[0;32m---> 93\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m     94\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_start_reading()\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:90\u001b[0m, in \u001b[0;36mBaseEventLoop.__init__\u001b[0;34m(self, transport_type, *args)\u001b[0m\n\u001b[1;32m     88\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_init()\n\u001b[1;32m     89\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 90\u001b[0m     \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m_connect_\u001b[39;49m\u001b[38;5;132;43;01m{}\u001b[39;49;00m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mformat\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransport_type\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     91\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     92\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclose()\n",
      "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/asyncio.py:96\u001b[0m, in \u001b[0;36mAsyncioEventLoop._connect_socket\u001b[0;34m(self, path)\u001b[0m\n\u001b[1;32m     94\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     95\u001b[0m     coroutine \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_loop\u001b[38;5;241m.\u001b[39mcreate_unix_connection(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fact, path)\n\u001b[0;32m---> 96\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_loop\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_until_complete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcoroutine\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:622\u001b[0m, in \u001b[0;36mBaseEventLoop.run_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    611\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Run until the Future is done.\u001b[39;00m\n\u001b[1;32m    612\u001b[0m \n\u001b[1;32m    613\u001b[0m \u001b[38;5;124;03mIf the argument is a coroutine, it is wrapped in a Task.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    619\u001b[0m \u001b[38;5;124;03mReturn the Future's result, or raise its exception.\u001b[39;00m\n\u001b[1;32m    620\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    621\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_closed()\n\u001b[0;32m--> 622\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_check_running\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    624\u001b[0m new_task \u001b[38;5;241m=\u001b[39m \u001b[38;5;129;01mnot\u001b[39;00m futures\u001b[38;5;241m.\u001b[39misfuture(future)\n\u001b[1;32m    625\u001b[0m future \u001b[38;5;241m=\u001b[39m tasks\u001b[38;5;241m.\u001b[39mensure_future(future, loop\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "File \u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:584\u001b[0m, in \u001b[0;36mBaseEventLoop._check_running\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    582\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mThis event loop is already running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m    583\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m events\u001b[38;5;241m.\u001b[39m_get_running_loop() \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 584\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m    585\u001b[0m         \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mCannot run the event loop while another loop is running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Cannot run the event loop while another loop is running"
     ]
    }
   ],
   "source": [
    "%pip install pynvim\n",
    "from pynvim import attach\n",
    "blocker = input(\"!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\")\n",
    "nvim = attach('socket', path='/tmp/nvim')\n",
    "nvim.command('vsplit')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

[justinmk]

Don't have an answer to this unfortunately. Known issue: #489

Possible workaround: #326 (comment)

[beyarkay]

Thanks for the swift reply

[wookayin]

A workaround is to use nest_asyncio. It is not a perfect solution as the behavior of nested asyncio loop can be quite unintuitive and strange, but works at least minimally:

[1]: import nest_asyncio
     nest_asyncio.apply()

[2]: from pynvim import attach
     nvim = attach('socket', path='/tmp/nvim.sock')

[3]: nvim.current.buffer[:] = 'foo'