python: Enhance syntax highlighting for type hints

[ClanEver]

Release Notes:

• Python: Improved syntax highlighting for type hints.

Before

After

---

Why manual recursion?

• Due to tree-sitter grammar not supporting recursion in query (Recursive nodes are difficult to match in queries tree-sitter-grammars/tree-sitter-lua#24), currently only manual recursion is possible (refer to https://github.com/projekt0n/github-nvim-theme/pull/250/files).

Unable to highlight when simple structures appear before complex structures, example:

def t() -> str | dict[int, dict[int, dict[int, str]]]:
    pass

Because complex structures are parsed as subscript rather than generic_type by tree-sitter in this case ☹

Related:

• Python Syntax Highlighting for t | None is incorrect #14715
• Union Type (Python Doc)
• Type parameter lists (Python Doc)

[maxdeviant]

@maxbrunsfeld Do you have thoughts on this approach?

[maxbrunsfeld]

I don't think we should land this code. I get that it's working around a limitation in the query system right now, but I think we just need to fix that limitation.

I worry this is going to have a negative performance impact, and it's also just hard to understand and maintain.

[maxbrunsfeld]

@ClanEver Can you back out the one complex pattern from the query (sorry that it's not possible to highlight union types cleanly with our current scheme) so that we can land the other changes in the query?

[ClanEver]

@maxbrunsfeld Would it be acceptable to reduce the depth to 3?

; Union type X | Y (up to 3 types)
(type
  (binary_operator
    left: [
      (binary_operator
        left:  (_) @type
        right: (_) @type
      ) @type
      (_) @type
    ]
    right: (_) @type
  ) @type
)

[maxdeviant]

Thank you!

I removed the union type highlighting until we can come up with a better solution.