simplify remote plugins, massively

[justinmk]

Problem

The "remote plugin" model currently is too complex, and 99% of the complexity provides very little benefit.

• Lifecycle is complicated, unnecessarily hard to test, troubleshoot, and explain to plugin authors.
• Every "remote plugin host" must implement 1000s of lines of code to solve the same problem in every API client. This is a pointless waste of time.
  • Example: all of the code below is just for the "remote plugin" implementation in:
    • node-client:
      • neovim/src/plugin/
      • decorators/src/plugin/
      • neovim/src/host/
    • go-client:
      • nvim/plugin/
  • The benefits of the extra code are
    1. "multi-tenancy" (one node process for all node rplugins).
    2. "decorators" can be used in the remote module to define Nvim commands/autocmds.
• Too many moving parts, too many concepts.
  • why is a "plugin host" needed?
  • why can't I just call vim.rplugin('node', 'path/to/index.js'), then call functions defined in the remote process?
  • why do I need "decorators" (@pynvim.plugin) ? why can't I just use Lua to define commands/events in plugin/foo.lua?
    • @pynvim.command('MyCommand', …) vs Lua vim.api.nvim_create_user_command('MyCommand', function(...) vim.rpcrequest('MyCommand', ...))
• :UpdateRemotePlugins and the "manifest" are extra state that must be resolved, refreshed, and healthchecked.

Solution

• Reposition "remote plugins" as "remote modules". For example, a Node.js "remote plugin" is just a Node.js module that (1) imports "neovim", (2) can be started as a Nvim RPC client, and (3) handles requests from Nvim.
  • Users no longer need to globally install neovim-node-host.
  • Client defines RPC method handlers via setHandler().
  • Client calls setup() (placeholder name) which attaches and calls nvim_set_client_info() with the methods defined by setHandler().
• Eliminate the "host" / "client" distinction. The client is the host (it can handle requests from Nvim).
• Drop "multi tenancy". Each plugin starts its own client ("host"). Plugins do not share a "host".
• Drop host-specific "sugar" for registering Nvim-side commands/autocmds (aka "bindings"). Plugins define commands/autocmds in Lua, like any other Nvim plugin. The commands call methods in the remote module.

Implementation

Implementation details

From doc/remote_plugin.txt:

Plugin hosts ... take care of most boilerplate involved in defining commands, autocmds, and functions implemented over |RPC| connections. Hosts are loaded only when one of their registered plugins require it.

We can address the above use-cases as follows:

1. Remote-plugins are just plain Lua plugins (plugin/foo.lua) that start API clients and call functions on the client.
   • Eliminates all the unnecessary complexity of trying to find foo.js/foo.py/… plugin "mains".
2. Make it easy from Lua to start any API client and treat it as a "remote host".
   • One (API client) process per plugin.
     • Future: Consider "sharing" client processes in the future. Out of scope for now.
   • Eliminates all the redundant, per-platform (js/py/…) impls that implement a "remote host".
   • Drop the concept of a "plugin host". We only need plain old API clients, not a "plugin host".
   • Nvim Lua stdlib will provide (once—not in every API client!) an ergonomic interface to:
     1. start a "known" API client
     2. allow the caller to specify the path to the remote-plugin "module" or "main.
     3. define commands/autocmds/functions that call remote functions.
        • "Decorators" are not need for this! If it's painful to do this in Lua, fix that once, in Nvim instead of "fixing" it N times in every API client!
   • Examples (compare to :help remote-plugin-example):
     • go-client example
     • pynvim example:
       "Remote" Python code:

       import pynvim
       
       def main():
           pynvim.setHandler('command_handler', command_handler)
           pynvim.setHandler('autocmd_handler', autocmd_handler)
           pynvim.setHandler('function_handler', function_handler)
           # (placeholder name, TBD) 
           # Attaches and calls `nvim_set_client_info()` with the `methods` defined by `setHandler()`.
           pynvim.setup()
       main()

       "Local" Lua code:

       local rplugin = vim.rplugin('python', 'path/to/init.py')
       
       vim.api.nvim_create_user_command('Cmd',
         function(args)
           vim.rpcrequest(rplugin.chan_id, 'command_handler', args.args, args.range)
         end,
         { nargs = '*' })
       vim.api.nvim_create_autocmd({'BufEnter'}, {
         pattern = {"*.c", "*.h"},
         callback = function(ev)
           vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
         end
       })
       function my_func(...)
         vim.rpcrequest(rplugin.chan_id, 'function_handler', {...})
       end

3. With the above design...
   • ELIMINATES:
     • a fuckton of redundant documentation explaining crap like decorators, the "sync" flag, Remote plugin manifest, remote#host#RegisterPlugin, "bootstrapping" details.
     • a fuckton of code: neovim/src/plugin decorators/src/plugin neovim/src/host
     • rplugin/node/ runtimepath directory
     • "registration", :UpdateRemotePlugins, rplugin "manifest"
     • remote#host#PluginsForHost
     • NvimPlugin.registerFunction, NvimPlugin.registerAutocmd, NvimPlugin.registerCommand
   • STILL NEEDED:
     • instead of NvimPlugin, the remote plugin uses the exact same NvimClient type that is returned by attach().
       A remote plugin is just a library that operates on the same old NvimClient that any other API client operates on.
     • provider#Poll()
     • detect()
     • require()
   • GAINS:
     • The "bootstrapping" is now extremely obvioius and the plugin implementor fully controls when to call vim.rplugin() from Lua.
     • vim.rplugin() loads the plugin "main" and returns a channel:
       1. find the platform interpreter (node, python, …)
       2. start the interpreter with a stdio RPC channel, targeting the plugin "main" file.
          • The plugin "main" file is expected to attach() to stdio, and use the resulting NvimClient to serve requests.
       3. calls provider#Poll() until success.
       4. returns a channel id.
     • NO sugar for creating commands/functions/autocmds that connect to the remote plugin.
       If creating commands/functions/autocmds is cumbersome we should fix that IN GENERAL, not only for remote plugins.

       vim.api.nvim_create_autocmd({'BufEnter'}, {
         pattern = {"*.c", "*.h"},
         callback = function(ev)
           vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
         end
       })
       

FAQ

• How does a plugin avoid loading the interpreter (slow) on startup?
  • To "share" an API client we could add an optional ns (namespace) parameter to vim.rplugin(), then it could be called on-demand and re-uses the existing channel if found.
    • This doesn't need to be solved right now, can just use jobstart().

Related

• The above proposal has similarities to nvim-yarp cc @roxma
  • But unlike nvim-yarp, we can go further and eliminate the concept of "registering" anything.

Work plan

• Add a util function that migrates legacy plugins using their "specs".
• Deprecate :UpdateRemotePlugins.
  • point to :help remote-plugin-migrate
  • ask user to report legacy plugins that haven't migrated yet: Which plugins use :UpdateRemotePlugins ? #29270
• Update the core API clients (implement addHandler(); setup() in the client automatically calls nvim_set_client_info() with the methods defined by addHandler(); let jobstart() invoke the module directly):
  • go-client simplify remote plugins go-client#167
  • node-client WIP: redesign "remote plugins", eliminate "host" concept node-client#344
  • pynvim simplify remote plugins pynvim#567
  • ruby client simplify remote plugins neovim-ruby#108
• Update handling of g:node_host_prog, so it can point to node
  • Transitional phase: neovim-node-host will continue to be accepted; the path to node will be derived by inspecting the shebang in neovim-node-host.
• Update handling of g:ruby_host_prog, so it can point to ruby
  • Transitional phase: neovim-ruby-host will continue to be accepted; the path to ruby will be derived by inspecting the shebang in neovim-ruby-host.
• Update :checkhealth.
• Update :help remote-plugin.
• (Nvim 0.12) Remove old rplugin-related code
• (Nvim 0.12) Require g:node_host_prog to point to node, remove support for neovim-node-host.
• (Nvim 0.12) Require g:ruby_host_prog to point to ruby, remove support for neovim-ruby-host.

[tarruda]

Hi @justinmk. Following up from your comment in a issue I was subscribed to.

I will try to answer some of your questions, but please take it with a grain of salt as it has been 8 years since I worked on Neovim:

Lifecycle is complicated, unnecessarily hard to test, troubleshoot, and explain to plugin authors.

Can you elaborate on this one? I don't remember and couldn't find anything in the docs.

Every "remote plugin host" must implement a 1000s of lines of code to solve the same problem in every API client. This is a pointless waste of time.
why is a "plugin host" needed?

The idea of a plugin host is to implement the necessary boilerplate code to communicate with Neovim. There should not be any need to write more than one plugin host per language (at least that was my goal at the time).

why can't I just call vim.rplugin('node', 'path/to/index.js'), then call functions defined in the remote process?

How would path/to/index.js expose functions? Neovim uses msgpack-rpc to do it, so at very least there should be some infrastructure that allows it to expose functions via RPC, which should be done by the plugin host.

why do I need "decorators" (@pynvim.plugin) ?

IIRC, this decorator would be part of the python plugin host and an easy way to export functions to Neovim via RPC.

why can't I just use Lua to define commands/events in plugin/foo.lua?

Not sure if I understood this question, but if you are writing Lua plugins, then it is not a remote plugin.

In hindsight, the idea of remote plugins might not have been that great. While in theory the ability to write plugins in any programming language is cool, in practice what happened is that the community ended up embracing Lua as the standard language for plugins (At least I haven't seen any major plugins written in anything other than Lua and vimscript).

[justinmk]

The idea of a plugin host is to implement the necessary boilerplate code to communicate with Neovim.

I am distinguishing "API client" vs "plugin host".

• The "API client" part of node-client is here:
  • https://github.com/neovim/node-client/tree/master/packages/neovim/src/attach
  • https://github.com/neovim/node-client/tree/master/packages/neovim/src/api
• The "plugin host" part of node-client is here:
  • https://github.com/neovim/node-client/tree/master/packages/neovim/src/host
  • https://github.com/neovim/node-client/tree/master/packages/neovim/src/plugin

The "plugin host" is 99% just nice-to-have "sugar" that makes it possible to define Nvim commands/autocmds from the remote. None of that is really necessary, it's easy to do that from Lua (or we should improve that).

There should not be any need to write more than one plugin host per language

But implementing the "remote host" nice-to-have stuff in every language is a chore, and a source of extra bugs, and doubles the amount of code + docs needed in each API client.

How would path/to/index.js expose functions?

The remote host will need to handle rpcrequest() calls from Nvim and "dispatch" them. That will require some sort of initialization/declaration in the remote module. But that still avoids needing to register things in a manifest from the Nvim (client) side.

IIRC, this decorator would be part of the python plugin host and an easy way to export functions to Neovim via RPC.

Yes. Not worth the complexity and bugs.

Not sure if I understood this question, but if you are writing Lua plugins, then it is not a remote plugin.

I'm proposing to move the bindings logic to Lua:

• start and manage an API client(s), from Lua
• call remote functions, and setup remote handlers, from Lua

but the business logic stays in the remote module.

Currently, trying to set up bindings from the remote side is 99% of the complexity, but 1% of the value. The business logic (not bindings) is where remote plugins can be useful. Furthermore, with this simplified design, we get it nearly for free.

in practice what happened is that the community ended up embracing Lua

That's parially because creating a remote plugin is too complicated. This proposal fixes that. There are cases where I want to integrate with code that doesn't have a CLI but can be loaded as a node/python/etc module.

[justinmk]

Skeleton / proof of concept: neovim/node-client#344