Using typeshed related tools while playing nice with other tools in the python ecosystem.

[couling]

I'm rather stuck knowing which way to turn or which way to advise our team. The problem we're facing started with an error reported in mypy which after some digging, was ultimately triggered by typeshed definitions. The typeshed definitions are technically correct but conflict with rational interpretations made by tools that don't use typeshed.

In particular, many tools that don't use typeshed assume that decorators have no effect on function signature. While technically incorrect, this assumption is at least rational and pragmatic.

Specific example @contextlib.contextmanager

This decorator changes the function signature, so while the internal method returns an Iterable or Generator, the wrapped method returns a ContextManager. Mypy uses typeshed's definition and so expects the method to be hinted as an Iterable, raising an error otherwise.

But many tools do not use typeshed, particularly tools for docstring interpretation like mkdocstrings-python. If we hint with Iterable in the source code, then that's exactly what will be printed in our documentation and that is NOT correct. Worse other linters not using typeshed will raise warnings because they do not understand the function signature has changed.

It leaves us stuck between a rock and a hard place, either we live with Mypi constantly warning us (disabling it completely for such a function), or we live with many other tools complaining, and we live with incorrect documentation.

General discussion

It seems that having separate type information in typeshed that's not in the origin source creates something of a mess, where it can conflict with reasonable interpretations from those tools that don't use typeshed. As much as it could be argued that all such tools should be using it, the fact is they don't.

For now, I think our decision is to disable Mypy warnings using # type: disable and deliberately typehint "wrong" in a way that static code analysers can more reliably interpret. I certainly don't want incorrect documentation.

Honestly I don't like this choice, but I don't see a practical way to proceed while the existence of typeshed creates a big split between different tools.

[JelleZijlstra]

I'm not sure why you are putting the blame on typeshed here: the root of the problem you're seeing is that the @contextmanager transforms the type of the generated function, and that would be true whether its types are defined in typeshed or inline in contextlib.py. The solution should be to make the tools that are behaving incorrectly somehow aware of the fact that @contextmanager transforms the signature, either by using typeshed, or by some other mechanism. (For example, perhaps the implementation of @contextmanager could set a fixed .__signature__ on the wrapper function.)

[couling]

That all suggests that we're unlikely to find any way out of this quagmire any time soon. As you describe, doing so is likely to be either too complicated or still flawed for many simpler tools.

In a way, those words have actually given me more sympathy with other developers who, no doubt knowingly, implemented their tools in a way that was "wrong" but at least workable.

That makes me sad too. If extraction of a functions's return type is too complicated for docstring generators to get right, then it hints at python's type hinting being somehow not quite fit for purpose.

Sorry. I guess I'm being provocative in my lament.

[erictraut]

I think your options are:

1. Accept the limitations of this tool. It sounds like it works in most cases but falls short in a few.
2. Work around the limitations of this tool, e.g. by doing special-case post-processing of its output to fix the few cases that it doesn't handle correctly.
3. Submit improvements to the tool, assuming it is open-sourced. This benefits everyone who uses it.
4. Find a better tool that doesn't have these limitations. This is the beauty of an open ecosystem. If there is sufficient need, someone will typically step up and solve the problem in a robust way.

[gandhis1]

Notably pylint can have the same issue and has a setting called signature-mutators to allow people to silence false positives.

[couling]

@erictraut testing your theory on point 4, if you have any suggestions for a docstring to markdown exporter that is typeshed based or compatible I'd love to give it a try. I'm totally open to changing our toolset. Though we came up with surprisingly few viable alternatives for this task.

[hmc-cs-mdrissi]

There is one more option a documentation library could take. Allowlist a couple decorators and keep them as part of documentation. Having documentation include context manager is probably more straight forward than having to mutate return type. For a couple like contextmanager the decorator is probably easier to understand then the real return type.

I’m unaware of any library that has a convenient public api for simple type inference questions. Like you could in theory run mypy/pyright/other type checkers in a subprocess but a simple question of give me the type signature of certain function at runtime gets tricky as soon as you can’t just use get_type_hints/annotations directly.

edit: I realized that documentation which shows @classmethod must be special casing that decorator. So extending that special case to more known mutator decorators or an allowlist could work.

[couling]

After giving this a lot more thought I keep coming back to two things:

Firstly pythons's ecosystem¹ has not converged² on one single interpretation of type hinting for these cases. Yes I know one way might be "right" and the other "wrong" but that's beside the point. It's unlikely that a common interpretation will be forthcoming in the near future because of the complexity that would involve for some otherwise simple tools.

Secondly PEP 484 has some very pertinent words for these situations which are even highlight in bold in the original text:

It should also be emphasised that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.

Therefore it seems most pragmatic to me to simply NOT hint critically problematic functions and instead ensure the documentation is extra clear. If this involves disabling Mypy for those functions then so be it.

This ensures that functions are not incorrectly hinted in any interpretation and are ultimately interpreted the same in all.

The hope is that someday the ecosystem will converge and hints can be added back at that time.

---

• ¹ Software not people
• ² yet

[erictraut]

Firstly pythons's ecosystem has not converged on one single interpretation of type hinting for these cases

I think that's incorrect. The ecosystem has converged on a single interpretation for how type annotations are meant to be interpreted. Some tools may contain bugs or limitations where they do not honor this interpretation in certain circumstances. If you were to ask the maintainers of mkdocstrings-python whether they believe that type annotations should be interpreted differently than how mypy (and PEP 484) indicate they should, I'd be very surprised if they said yes. So I don't think there's any confusion about how type annotations should be interpreted. This is simply a case of a tool that has some limitations. Most tools do.

I'll add one more option to my list above:
5. Work around the tool's limitation by avoiding the use of decorators that cause problems. The @contextlib.contextmanager decorator is a convenience function. You can easily replace it with your own implementation that performs the same function. By doing so, you can annotate the outer wrapper in a way that works around the limitation of mkdocstrings-python.

[couling]

@erictraut i don't think it's fair to claim that the ecosystem has converged when so many tools remain unable to implement the design and those that don't have such a consistent way to do it "wrong".

the points we've discussed so far at not mere typo style bugs, they are limitations caused by a design so complicated that many developers are unable to work with it.

Developers of the ecosystem might want it to converge a particular way, but at this time they have been unable to make this happen.

I remain loathed to use a typing system that is explicitly not mandatory even by convention in places where the tooling (software not people) cannot agree on how to interpret it.

[pawamoy]

Interesting discussion, thanks! I'm the author of mkdocstrings 🙂
I think I completely agree with @erictraut. It's true that mkdocstrings-python (actually Griffe, the underlying tool that parses Python code) does not use typeshed to provide correct annotations for functions decorated with signature-changing decorators such as contextlib.contextmanager. To be honest, I just never thought about it. But I'm totally open to bring that kind of support into Griffe, whether it's by actually using typeshed some way or another, or simply by special-casing/post-processing specific things, one by one as users request them. So if you feel like it @couling, please do open tickets on our bugtrackers 👍