docs/automation: include luv meta definitions files

[altermo]

Problem

Currently, there's no luv _meta file for LSP annotations, and the most used lua LSP is planning to drop the builtin luv meta file, which would require users to download a separate luv meta library.

Expected behavior

Have the luv meta-file library built in.

Downside

Adding a large meta library such as luv to the meta files would increase LSP load times.

[justinmk]

Thanks for raising this.

With the VS Code addon manager releasing soon, we have decided to remove the built-in libraries that ship with the language server

Sigh.

I am currently investigating how we can make using and sharing definitions files even easier. Ideally, the language server will be able to install and manage addons defined in a setting by simply providing the git clone URLs. This would hopefully be cross-platform and super easy for all users.

Hopefully this will be implemented, then users (or nvim-lspconfig) can point to the metafile URL. Unless this requires vscode?

[clason]

Unless this requires vscode?

Yes.

[clason]

I'm vaguely in favor of this, since luv is our main dependency and core infrastructure. It would also tie in to autogenerating luvref.txt -- maybe luvit themselves would be open to maintaining their own _meta file? I certainly don't think it's reasonable for us to maintain our own _meta_ file.

(Other directly exposed dependencies like lpeg and msgpack have diminishing returns.)

[justinmk]

Even if the "VS Code addon manager" doesn't help us, it seems like the metafiles will be maintained by LuaLS and available via some URL, which we could then consume via bump_deps.lua (automated vendoring) or a user command (so we don't need to vendor).

[lewis6991]

This is still an unfinished thread in my lua type annotations crusade: luvit/luv#632

[clason]

https://github.com/Bilal2453/luvit-meta

(in particular https://github.com/Bilal2453/luvit-meta/blob/main/library/uv.lua)

[wookayin]

First of all, this has less thing to do with lsp but is all about documentation and automation, so the title could be changed as docs: instead of LSP:.

I am investigating how we can include luv (vim.uv) meta files, and also generate or maintain vimdoc luvref.txt. As already discussed earlier in this issue, this is quite a complex one until luvit/luv#632 is resolved. I raise several discussion points as follows.

What would be the gold (upstream) "source" for libuv documentation that neovim can pull?

We have two choices.

(1) https://github.com/luvit/luv/blob/master/docs.md is the docs maintained and kept up-to-date with the actual libuv implementation.

• Pros: This doc is supposed to be always up-to-date with the actual libuv implementation, so the docs and the library will remain synchronized and compatible with each other.
• Problem: it's not machine-readable (see Maintain documentation in a machine readable format luvit/luv#632).
• Possible solution: Manual processing/edit is needed. Essentially the same (duplicate) efforts as in (2) done on the neovim side.
• Possible solution: Or we can write a scripted generator, but this would be of too much work and maintenance burden (lots of corner cases will bother us).

(2) https://github.com/Bilal2453/luvit-meta (maintained by @Bilal2453)

• Pros: Already working solution for Lua-LS. @folke's neodev also bundles exactly the same file uv.lua from this repository. If neovim pulls this, neovim doesn't need to own or maintain this.
• Problem (maintenance): This is maintained by "manually editing" the meta file, following the changes time to time.
• Problem (maintenance): It's not clearly tracked or mentioned that the current HEAD version is compatible with or catching up to which luv version. Therefore, we cannot guarantee that the meta file exactly matches the libuv version we include as the CMakeFile dependency. This could be improved in coordination/collaboration with @Bilal2453, but since this is basically a 3rd-party repository we don't have much control.
• Problem (technical): neovim's vimdoc generator (lua2dox + gen_vimdoc) does not like the format. This is mainly because of two problems (1) overload functions and (2) alias types, see below.

(3) What lua-ls currently ships as meta/3rd

• https://github.com/LuaLS/lua-language-server/tree/master/meta/3rd -> https://github.com/LuaCATS/luv/blob/3615eb12c94a7cfa7184b8488cf908abb5e94c9c/library/uv.lua
• Pretty outdated, 2 years old; this is going to be removed from lua-ls as @altermo already pointed out, so we don't need to consider this one.

Technical Problems on doc generator on the uv.lua from Bilal2453/luvit-meta

Note that the current meta file uv.lua works just well for Lua LSP. But the problem is when neovim wants to generate vimdoc (luvref.txt) from this meta file.

(1) Overload functions with different params/returns signature, e.g., uv.fs_open().

SOLUTION: This can be addressed on the neovim side without requiring the meta file to change its style: I managed to improve gen_vimdoc and lua2dox to handle overloaded functions.

Example of generated *uv.fs_open()* vimdoc

uv.fs_open({path}, {flags}, {mode}, {callback})                 *uv.fs_open()*
uv.fs_open({path}, {flags}, {mode})                                           
    Equivalent to `open(2)`. See below for available access `flags`.

    Note: On Windows, libuv uses `CreateFileW` and thus the file is always
    opened in binary mode. Because of this, the `O_BINARY` and `O_TEXT` flags
    are not supported.

    Parameters: ~
      • {path}      (`string`)
      • {flags}     (`uv.aliases.fs_access_flags|integer`)
      • {mode}      (`integer`)
      • {callback}  (`fun(err: string, fd: integer?)?`)

    Return: ~
        (`uv_fs_t`)

    Parameters: ~
      • {path}   (`string`)
      • {flags}  (`uv.aliases.fs_access_flags|integer`)
      • {mode}   (`integer`)

    Return: ~
        (`integer? fd, string? err_name, string? err_msg`)

Admittedly it's a bit ugly to show [Parameters & Returns] twice. Merging them into one is non-trivial because parameter/return types are not necessarily the same.

(2) Parameters for complex table options are translated into alias lua types, e.g.

• uv.aliases.getaddrinfo_hint for uv.getaddrinfo
• and their human-readable explanation on the parameters are intentionally and manually removed from the docstring.

While this might be useful for Lua_LS, this causes some problems when generating vimdocs (luvref.txt) because our doc generator does not unroll the alias types. Personally I think what is written in the markdown doc is more useful for vimdoc; but as we need to generate vimdoc automatically from this meta file, the docs included in the current version of the meta file won't be enough.

• OPTION 1: Update the meta file uv.lua to include this back (the simplest solution, but requires a major re-work on the meta file).
• OPTION 2: Just don't show them. This would be a regression in the luvref.txt (Example below)

Example for OPTION 2

-uv.getaddrinfo({host}, {service} [, {hints} [, {callback}]])  *uv.getaddrinfo()*
-
-                Parameters:
-                - `host`: `string` or `nil`
-                - `service`: `string` or `nil`
-                - `hints`: `table` or `nil`
-                  - `family`: `string` or `integer` or `nil`
-                  - `socktype`: `string` or `integer` or `nil`
-                  - `protocol`: `string` or `integer` or `nil`
-                  - `addrconfig`: `boolean` or `nil`
-                  - `v4mapped`: `boolean` or `nil`
-                  - `all`: `boolean` or `nil`
-                  - `numerichost`: `boolean` or `nil`
-                  - `passive`: `boolean` or `nil`
-                  - `numericserv`: `boolean` or `nil`
-                  - `canonname`: `boolean` or `nil`
-                - `callback`: `callable` (async version) or `nil` (sync
-                  version)
-                  - `err`: `nil` or `string`
-                  - `addresses`: `table` or `nil` (see below)
+
+                                                           *uv.getaddrinfo()*
+uv.getaddrinfo({host}, {service}, {hints}, {callback})
+uv.getaddrinfo({host}, {service}, {hints})                                    

     Equivalent to `getaddrinfo(3)`. Either `host` or `service` may be `nil`
     but not both.

-                Valid hint strings for the keys that take a string:
-                - `family`: `"unix"`, `"inet"`, `"inet6"`, `"ipx"`,
-                  `"netlink"`, `"x25"`, `"ax25"`, `"atmpvc"`, `"appletalk"`,
-                  or `"packet"`
-                - `socktype`: `"stream"`, `"dgram"`, `"raw"`, `"rdm"`, or
-                  `"seqpacket"`
-                - `protocol`: will be looked up using the `getprotobyname(3)`
-                  function (examples: `"ip"`, `"icmp"`, `"tcp"`, `"udp"`, etc)-
+
+    Parameters: ~
+      • {host}      (`string?`)
+      • {service}   (`string?`)
+      • {hints}     (`uv.aliases.getaddrinfo_hint?`)
+      • {callback}  (`fun(err?: string, addresses?:
+                    uv.aliases.getaddrinfo_rtn)`)

-                Returns (sync version): `table` or `fail`
-                - `[1, 2, 3, ..., n]` : `table`
-                  - `addr` : `string`
-                  - `family` : `string`
-                  - `port` : `integer` or `nil`
-                  - `socktype` : `string`
-                  - `protocol` : `string`
-                  - `canonname` : `string` or `nil`
- 
-               Returns (async version): `uv_getaddrinfo_t userdata` or `fail`
+   Return: ~
+       (`uv_getaddrinfo_t`)

Users will have no idea about uv.aliases.getaddrinfo_hint, uv_getaddrinfo_t, etc.

• OPTION 3: Auto-generate vimdoc from the meta file, but resolve the alias type (uv.aliases.getaddrinfo_hint?) to generate a sub-list (family, socktype, etc.). This will require a non-trivial work on our documentation generator.

I can open a draft PR to show the preliminary work after some clean-up.

(3) Generating sections for vimdoc is also difficult because of the different doc structure (@section v.s. @defgroup) being assumed.

Decision to make:

• Should neovim "own" a meta file, or pull from Bilal2453/luvit-meta or directly from docs.md?

• Do we generate luvref.txt or maintain them manually (in addition to _meta/uv.lua)? How much difference between luvref.txt and docs.md we will allow?

• Or should we wait until Maintain documentation in a machine readable format luvit/luv#632 is done? (it seems that it won't be done in the very near future)

• An alternative possible approach is the opposite way; to generate (manually maintained) the meta file for vim.uv from luvref.txt, instead of generating luvref.txt from the meta files. This is the former approach used by @folke in neodev, but currently it was superseded by just fetching the uv.lua meta file. In neovim we don't generate meta files from doc files, so I don't think this is a very compelling approach.

Also, we should make sure that luv meta files and docs are up-to-date and properly synchronized (either manually or automatically) whenever libuv dependency is bumped. #22186

If we are fine with a "minimal viable" vimdoc that might be somewhat different from the current `luvref.txt or https://github.com/luvit/luv/blob/master/docs.md, we can choose the option to use @Bilal2453's one (with some patches contributed) and generate vimdoc from it.

[clason]

For the record, there's a variant of these options: We provide the upstream meta file (starting from one of the existing files, edited to suit us), in the hope that once it's there, upstream will maintain it and we can just pull it on any luv bump.

[lewis6991]

That might be the easiest thing tbh.

[Bilal2453]

Hey there, I am happy to help with this!

First for the mentioned problems in the approach of using luvit-meta/uv.lua, I am working on a more robust annotation layout that is more friendly to a doc-gen (see the weird @namespace and @section annotations), namely this would solve the problem of tracking as it will include the luv commit hash (and release) it annotates as well as separates the luv annotations into sections similar to the current markdown layout. But I am afraid that might not be the case for the other remaining problems, I don't believe I can change the overload style used, if I recall correctly I was forced to use it due to LSP limitations (though not sure if that is solved now).
I do happen to be maintaining a fair bit of the luv docs, and I make sure that whatever commit going to doc.md also goes to uv.lua, so far luvit-meta has an up to date coverage of it.

That being said, I probably wouldn't take either of the mentioned approaches, I had to manually add things in/out from annotations because the automatically generated ones from the markdown (which served as the base) were either inconsistent or incomplete for something like this, my real intention behind the luvit-meta project was not to actually provide types or annotations for Luvit and Luv, but to compensate for the lack of documentation and to eventually replace the upstream docs by converting the annotations into something more usable and easy to maintain (something like JSON files / Lua tables).
While my approach might sound like adding more stages than it is actually needed (which it is) this was done because it was a big project for only a single person (Luvit API is hugely undocumented), it felt like I would waste my time writing JSON (for example) only to get burned out eventually and achieve nothing, so I decided to go with the more painful route of writing the annotations first (which comes with its own cons) so that in case I do get burned out (which I eventually did, only 50-60% of luvit-meta is complete) so I at least end up with some achievement.

In my opinion, the approach that should be taken is more inline with what @clason suggested, work on luvit/luv#632 upstream, write a machine-readable luv docs, and use that to generate the annotations.
I started working on a doc-gen converting the annotations I have into an Intermediate form a while ago but quickly faced some challenges (for example, I faced a similar issue with @aliases) and got busy with other things.

[wookayin]

Hi @Bilal2453, thanks for your feedback and help! I look forward to working with you together to have a improved meta file (either on your repo or potentially contribute to upstream luvit) that can serve Neovim's needs and your needs. I can feel and relate the huge efforts and manual processes you've already gone through.

Regarding overload functions --- I think the current style of overload functions are fine; they follow standard LuaCATS annotation style and it's something that neovim's build process can adapt to.

Since it seems that you have some preliminary works done towards luvit/luv#632, I'd be happy to involved if there's anything I can help you with. I will also keep you involved once I start to make any further steps (currently it's more like a planning/design phase for me).

[Bilal2453]

A tiny bit of progress, I will be working on getting something working over at luv-docgen.
Currently I have a working lexer (which was 10 lines of LPeg grammar) and will be working on the parser next (described in the repo), at the end of it we should have a machine-readable structure that describes the docs well enough for both a markdown docgen and LuaCAT annotations.

I am not currently sure if I need help with something coding wise. Will probably need more opinions about the potential problems will be facing, and the Lua table structure of the parser (both described in the repo). I will be posting further updates over at luvit/luv#632, thanks for everyone's help!

[wookayin]

@Bilal2453 Sounds great, I will have a look and give some suggestions.

Also take a look at the LPeg grammar neovim uses https://github.com/neovim/neovim/blob/master/src/nvim/generators/luacats_grammar.lua, written by @lewis6991 (#26967). This handles only @params and @returns at the moment for the purpose of vimdoc generation, but your work will be a more general solution; which we look forward to using once it becomes mature.

[Bilal2453]

Judging by how it is going, I am slightly doubting if my parser would really be any better for a more general solution (at least outside of my luv definitions). Doing things right (like accounting for invalid inputs, etc) requires a fair bit of time and I am a bit in a rush as I want to finish this in a day or two (hopefully) before I am busy with work, so I am mostly using Lua's string matching with assumption about what the usage of an annotation looks like.
Of course for generating the Luv vimdocs, the generated data.lua should do.

For a long term solution why not use the LuaLS parser itself? It will guarantee that your parser is always doing what it is supposed to do, and will be really easy to maintain it for any upstream changes (like adding new annotations, syntax etc), the only disadvantages of that is A) you have to depend on code outside of your repo and B) the parser is meant for LSP, its Lua structure output is very big and complicated (which is the only reason I didn't use it instead).

[clason]

LuaLS is not reliable as a dependency, and as a black box unsuitable.

[Bilal2453]

I meant using the LuaLS parser itself, not the binary of the language server, does that apply to it too? From what I've seen it seems like it would be reliable, for instance, you need this parser directory + utility.lua, and lpeglabel, parsing definitions would then be like:

local parser = require('parser')
local state = parser.compile(input, 'Lua', 'Lua 5.4') -- the version shouldn't really matter

parser.luadoc(state) -- parse docs (definitions)

-- now accessible with state.ast.docs/etc

Though you need then to convert this structure into something actually usable, accessing the state directly is painful.

[clason]

Yeah, the whole project is not reliable (for this purpose). Rolling our own with Lpeg is the right way in my view.

[wookayin]

Good point, it feels like we're kind of reinventing a wheel, but Neovim's LuaLS annotation use is not 100% strictly compliant with the LuaCATS (for historical reasons), which also changes gradually over time. We have some non-standard annotations that LuaLS may parse differently. Another unresolved issue that I know of is: LuaLS/lua-language-server#2333 -- docs for @param are not parsed correctly and we don't have much control inside the black box.

[Bilal2453]

Any annotation tag that isn't recognized by the parser is treated as a normal comment, so it should still be possible to handle it in a later step.

For that bug and any future ones, I think it would be neat if we fix those upstream (I might look into, doesn't sound terribly hard to support, famous last words), descriptions are the most complicated thing to parse (from my experience at least), especially ones that don't explicitly use #, because it requires full support to any possibly acceptable value, and you can only start actually parsing the description once you have fully finished parsing what is before it, which makes it prune to a lot of errors and any new syntax/feature can easily break it. I wish if the description syntax that didn't use # simply didn't exists!

Nonetheless, will see what happens with mine and you could decide from there.