Improve import time of various stdlib modules

[layday]

Feature or enhancement

Proposal:

Following on from #109653, further improvements can be made to import times.

Links to previous discussion of this feature:

https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173

For example:

importlib.metadata is often used for tasks that need to happen at import, e.g. to enumerate/load entry point plug-ins, so it might be worth seeing if we can cut down its own import time a bit more.

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless. Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

Linked PRs

• gh-118761: Improve import time of pprint #122725
• gh-118761: Speedup pathlib import by deferring shutil #123520
• gh-121423: Improve import time of socket by writing socket.errorTab as a constant and lazy import modules #121424
• gh-118761: Improve import time of mimetypes #126979
• gh-118761: improve import time for pickle #128732
• gh-118761: substitute re import in base64.b16decode for a more efficient alternative #128736
• gh-118761: improve import time for secrets #128738
• gh-118761: Improve import time for csv #128858
• gh-118761: Reduce import time of gettext.py by delaying re import #128898
• gh-118761: improve optparse import time by delaying textwrap import #128899
• gh-118761: Improve import time of tomllib #128907
• gh-118761: Improve import time for pstats and zipfile by removing imports to typing #128981
• gh-118761: Improve import time of sqlite3 #129118
• gh-118761: Improve import time of subprocess #129427
• gh-118761: Always lazy import warnings in threading #129428
• [3.13] gh-118761: Improve import time of subprocess (GH-129427) #129447
• [3.12] gh-118761: Improve import time of subprocess (GH-129427) #129448
• gh-118761: Improve import time by lazy import of warnings #129765
• gh-118761: Improve import time by lazy import of traceback #129811
• gh-118761: Always lazy import re in locale #129860
• gh-118761: Improve import time of dataclasses #129925
• gh-118761: Improve import time of cmd module #130056
• gh-118761: Revert "Improve import time of subprocess (GH-129427)" #130201
• [3.13] gh-118761: Revert "Improve import time of subprocess (GH-129427)" (GH-130201) #130204
• [3.12] gh-118761: Revert "Improve import time of subprocess (GH-129427)" (GH-130201) #130205
• gh-118761: Improve import time of sqlite3 #131796
• gh-118761: Optimise import time for ast #131953
• gh-118761: Optimise import time for textwrap #131956
• gh-118761: Fix star-import of ast #132024
• gh-118761: Fix star-import of ast (alternative) #132025
• gh-118761: Improve import time of annotationlib #132028
• gh-118761: Use enum._simple_enum for annotationlib.Format #132031
• gh-118761: Cover the import time optimisations in What's New #132035
• gh-118761: Optimise import time for shlex #132036
• gh-118761: Optimise import time for string #132037
• gh-118761: email.quoprimime removing re import #132046
• gh-118761: Defer import of functools in annotationlib #132059
• gh-118761: Lazily import annotationlib in typing #132060
• GH-118761: Expose more core interpreter types in _types #132103

[hugovk]

@layday Is it okay if I repurpose this issue as an "Improve import time of various stdlib modules" like #109653 but for 3.14?

I've got some pprint improvements, and if we have importlib.metadata and some others, we can group them under the same umbrella issue like last time.

[layday]

Sure!

[danielhollas]

I've opened a PR over at the importlib_metadata repo that avoids importing inspect. python/importlib_metadata#499

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases.
Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

@layday were you planning on tackling these?

It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method

This seems to be solved on main, importlib.abc no longer imports importlib.resources.

[danielhollas]

I've opened a PR over at the importlib_metadata repo that avoids importing inspect. python/importlib_metadata#499

This has been merged and released in version 8.4 of importlib_metadata 🎉

importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless. Some other less frequently-used imports which are only accessed once or twice, such as json, could also be tucked away in their calling functions.

I've submitted python/importlib_metadata#502 that defers zip import, and python/importlib_metadata#503 which defers json and platform.

[picnixz]

(removing the 3.14 label since features always target the main branch)

[hugovk]

Note for when documenting this in What's New in Python 3.14, can also include #128559 / #128560.

[vstinner]

@picnixz: I suggest to close this issue.

[AA-Turner]

I've created a PR for ast (#131953) and a draft PR using self-overwriting descriptors to optimise textwrap (#131956).

A

[TeamSpen210]

Since typing has a lot of different things defined, many distinct from each other, I wonder if it'd be worth it to either break it up internally or put code into functions, then define module __getattr__() to lazily construct groups of objects as necessary. Since often modules just want one or two things.

[ofek]

I haven't looked at the code but it would be a big win if that is possible since basically everything imports that nowadays and there is no way to avoid it.

[JelleZijlstra]

@TeamSpen210 something like that could be worthwhile but the maintainability cost might be pretty high. Also, there's a risk that programs that do use a lot of things from typing get slower because of the extra indirection.

There might also be interpreter-level things we can do to speed up specific operations that often happen at import time. For example, creating an empty class is about 70x slower than creating an empty function—maybe we can improve that.

[AA-Turner]

Continuing on ast optimisations, I get a ~10% speed-up ¹ by moving the deprecated classes (ast.slice onwards) into a __getattr__. Class creation is slow!

This does feel borderline, though. Any thoughts?

Footnotes

1. From 2273.2µs to 2052.1µs (cumulative); from 683.0µs to 539.8µs (self). ↩

[JelleZijlstra]

I looked at some profiling data and feel there's good opportunities for optimizing class creation if someone is interested in hacking on typeobject.c: see #132042 for details. Making all class creation faster should help a lot with importing large modules like typing.

[JelleZijlstra]

For typing, most of the remaining cost is now attributable to the imports of collections and functools. functools is unfortunately hard to avoid, because we need it in the _tp_cache decorator that gets executed at import time. collections is doable (by using the private _collections_abc module and some __getattr__ hacks), but functools also imports collections, so we don't currently gain anything by deferring the import in typing.