Walk up for all config files and handle precedence (#18482)

Follow up to #16965
Fixes #16070

Handles other mypy configuration files and handles precedence between
them. Also fixes few small things, like use in git worktrees

Author: hauntsaninja

Diff for: CHANGELOG.md

@@ -9,23 +9,16 @@ garbage collector.
 
 Contributed by Jukka Lehtosalo (PR [18306](https://github.com/python/mypy/pull/18306)).
 
-### Drop Support for Python 3.8
-
-Mypy no longer supports running with Python 3.8, which has reached end-of-life.
-When running mypy with Python 3.9+, it is still possible to type check code
-that needs to support Python 3.8 with the `--python-version 3.8` argument.
-Support for this will be dropped in the first half of 2025!
-
-Contributed by Marc Mueller (PR [17492](https://github.com/python/mypy/pull/17492)).
-
 ### Mypyc accelerated mypy wheels for aarch64
 
 Mypy can compile itself to C extension modules using mypyc. This makes mypy 3-5x faster
 than if mypy is interpreted with pure Python. We now build and upload mypyc accelerated
 mypy wheels for `manylinux_aarch64` to PyPI, making it easy for users on such platforms
 to realise this speedup.
 
-Contributed by Christian Bundy (PR [mypy_mypyc-wheels#76](https://github.com/mypyc/mypy_mypyc-wheels/pull/76))
+Contributed by Christian Bundy and Marc Mueller
+(PR [mypy_mypyc-wheels#76](https://github.com/mypyc/mypy_mypyc-wheels/pull/76),
+PR [mypy_mypyc-wheels#89](https://github.com/mypyc/mypy_mypyc-wheels/pull/89)).
 
 ### `--strict-bytes`
 
@@ -48,6 +41,16 @@ Contributed by Christoph Tyralla (PR [18180](https://github.com/python/mypy/pull
 (Speaking of partial types, another reminder that mypy plans on enabling `--local-partial-types`
 by default in **mypy 2.0**).
 
+### Better discovery of configuration files
+
+Mypy will now walk up the filesystem (up until a repository or file system root) to discover
+configuration files. See the
+[mypy configuration file documentation](https://mypy.readthedocs.io/en/stable/config_file.html)
+for more details.
+
+Contributed by Mikhail Shiryaev and Shantanu Jain
+(PR [16965](https://github.com/python/mypy/pull/16965), PR [18482](https://github.com/python/mypy/pull/18482)
+
 ### Better line numbers for decorators and slice expressions
 
 Mypy now uses more correct line numbers for decorators and slice expressions. In some cases, this
@@ -56,6 +59,15 @@ may necessitate changing the location of a `# type: ignore` comment.
 Contributed by Shantanu Jain (PR [18392](https://github.com/python/mypy/pull/18392),
 PR [18397](https://github.com/python/mypy/pull/18397)).
 
+### Drop Support for Python 3.8
+
+Mypy no longer supports running with Python 3.8, which has reached end-of-life.
+When running mypy with Python 3.9+, it is still possible to type check code
+that needs to support Python 3.8 with the `--python-version 3.8` argument.
+Support for this will be dropped in the first half of 2025!
+
+Contributed by Marc Mueller (PR [17492](https://github.com/python/mypy/pull/17492)).
+
 ## Mypy 1.14
 
 We’ve just uploaded mypy 1.14 to the Python Package Index ([PyPI](https://pypi.org/project/mypy/)).

Diff for: docs/source/config_file.rst

@@ -7,22 +7,30 @@ Mypy is very configurable. This is most useful when introducing typing to
 an existing codebase. See :ref:`existing-code` for concrete advice for
 that situation.
 
-Mypy supports reading configuration settings from a file with the following precedence order:
+Mypy supports reading configuration settings from a file. By default, mypy will
+discover configuration files by walking up the file system (up until the root of
+a repository or the root of the filesystem). In each directory, it will look for
+the following configuration files (in this order):
 
-    1. ``./mypy.ini``
-    2. ``./.mypy.ini``
-    3. ``./pyproject.toml``
-    4. ``./setup.cfg``
-    5. ``$XDG_CONFIG_HOME/mypy/config``
-    6. ``~/.config/mypy/config``
-    7. ``~/.mypy.ini``
+    1. ``mypy.ini``
+    2. ``.mypy.ini``
+    3. ``pyproject.toml`` (containing a ``[tool.mypy]`` section)
+    4. ``setup.cfg`` (containing a ``[mypy]`` section)
+
+If no configuration file is found by this method, mypy will then look for
+configuration files in the following locations (in this order):
+
+    1. ``$XDG_CONFIG_HOME/mypy/config``
+    2. ``~/.config/mypy/config``
+    3. ``~/.mypy.ini``
+
+The :option:`--config-file <mypy --config-file>` command-line flag has the
+highest precedence and must point towards a valid configuration file;
+otherwise mypy will report an error and exit. Without the command line option,
+mypy will look for configuration files in the precedence order above.
 
 It is important to understand that there is no merging of configuration
-files, as it would lead to ambiguity. The :option:`--config-file <mypy --config-file>`
-command-line flag has the highest precedence and
-must be correct; otherwise mypy will report an error and exit. Without the
-command line option, mypy will look for configuration files in the
-precedence order above.
+files, as it would lead to ambiguity.
 
 Most flags correspond closely to :ref:`command-line flags
 <command-line>` but there are some differences in flag names and some

Diff for: mypy/config_parser.py

@@ -15,7 +15,7 @@
 else:
     import tomli as tomllib
 
-from collections.abc import Iterable, Mapping, MutableMapping, Sequence
+from collections.abc import Mapping, MutableMapping, Sequence
 from typing import Any, Callable, Final, TextIO, Union
 from typing_extensions import TypeAlias as _TypeAlias
 
@@ -217,6 +217,72 @@ def split_commas(value: str) -> list[str]:
 )
 
 
+def _parse_individual_file(
+    config_file: str, stderr: TextIO | None = None
+) -> tuple[MutableMapping[str, Any], dict[str, _INI_PARSER_CALLABLE], str] | None:
+
+    if not os.path.exists(config_file):
+        return None
+
+    parser: MutableMapping[str, Any]
+    try:
+        if is_toml(config_file):
+            with open(config_file, "rb") as f:
+                toml_data = tomllib.load(f)
+            # Filter down to just mypy relevant toml keys
+            toml_data = toml_data.get("tool", {})
+            if "mypy" not in toml_data:
+                return None
+            toml_data = {"mypy": toml_data["mypy"]}
+            parser = destructure_overrides(toml_data)
+            config_types = toml_config_types
+        else:
+            parser = configparser.RawConfigParser()
+            parser.read(config_file)
+            config_types = ini_config_types
+
+    except (tomllib.TOMLDecodeError, configparser.Error, ConfigTOMLValueError) as err:
+        print(f"{config_file}: {err}", file=stderr)
+        return None
+
+    if os.path.basename(config_file) in defaults.SHARED_CONFIG_NAMES and "mypy" not in parser:
+        return None
+
+    return parser, config_types, config_file
+
+
+def _find_config_file(
+    stderr: TextIO | None = None,
+) -> tuple[MutableMapping[str, Any], dict[str, _INI_PARSER_CALLABLE], str] | None:
+
+    current_dir = os.path.abspath(os.getcwd())
+
+    while True:
+        for name in defaults.CONFIG_NAMES + defaults.SHARED_CONFIG_NAMES:
+            config_file = os.path.relpath(os.path.join(current_dir, name))
+            ret = _parse_individual_file(config_file, stderr)
+            if ret is None:
+                continue
+            return ret
+
+        if any(
+            os.path.exists(os.path.join(current_dir, cvs_root)) for cvs_root in (".git", ".hg")
+        ):
+            break
+        parent_dir = os.path.dirname(current_dir)
+        if parent_dir == current_dir:
+            break
+        current_dir = parent_dir
+
+    for config_file in defaults.USER_CONFIG_FILES:
+        ret = _parse_individual_file(config_file, stderr)
+        if ret is None:
+            continue
+        return ret
+
+    return None
+
+
 def parse_config_file(
     options: Options,
     set_strict_flags: Callable[[], None],
@@ -233,47 +299,20 @@ def parse_config_file(
     stdout = stdout or sys.stdout
     stderr = stderr or sys.stderr
 
-    if filename is not None:
-        config_files: tuple[str, ...] = (filename,)
-    else:
-        config_files_iter: Iterable[str] = map(os.path.expanduser, defaults.CONFIG_FILES)
-        config_files = tuple(config_files_iter)
-
-    config_parser = configparser.RawConfigParser()
-
-    for config_file in config_files:
-        if not os.path.exists(config_file):
-            continue
-        try:
-            if is_toml(config_file):
-                with open(config_file, "rb") as f:
-                    toml_data = tomllib.load(f)
-                # Filter down to just mypy relevant toml keys
-                toml_data = toml_data.get("tool", {})
-                if "mypy" not in toml_data:
-                    continue
-                toml_data = {"mypy": toml_data["mypy"]}
-                parser: MutableMapping[str, Any] = destructure_overrides(toml_data)
-                config_types = toml_config_types
-            else:
-                config_parser.read(config_file)
-                parser = config_parser
-                config_types = ini_config_types
-        except (tomllib.TOMLDecodeError, configparser.Error, ConfigTOMLValueError) as err:
-            print(f"{config_file}: {err}", file=stderr)
-        else:
-            if config_file in defaults.SHARED_CONFIG_FILES and "mypy" not in parser:
-                continue
-            file_read = config_file
-            options.config_file = file_read
-            break
-    else:
+    ret = (
+        _parse_individual_file(filename, stderr)
+        if filename is not None
+        else _find_config_file(stderr)
+    )
+    if ret is None:
         return
+    parser, config_types, file_read = ret
 
-    os.environ["MYPY_CONFIG_FILE_DIR"] = os.path.dirname(os.path.abspath(config_file))
+    options.config_file = file_read
+    os.environ["MYPY_CONFIG_FILE_DIR"] = os.path.dirname(os.path.abspath(file_read))
 
     if "mypy" not in parser:
-        if filename or file_read not in defaults.SHARED_CONFIG_FILES:
+        if filename or os.path.basename(file_read) not in defaults.SHARED_CONFIG_NAMES:
             print(f"{file_read}: No [mypy] section in config file", file=stderr)
     else:
         section = parser["mypy"]

Diff for: mypy/defaults.py

@@ -12,50 +12,15 @@
 # mypy, at least version PYTHON3_VERSION is needed.
 PYTHON3_VERSION_MIN: Final = (3, 8)  # Keep in sync with typeshed's python support
 
+CACHE_DIR: Final = ".mypy_cache"
 
-def find_pyproject() -> str:
-    """Search for file pyproject.toml in the parent directories recursively.
-
-    It resolves symlinks, so if there is any symlink up in the tree, it does not respect them
-
-    If the file is not found until the root of FS or repository, PYPROJECT_FILE is used
-    """
-
-    def is_root(current_dir: str) -> bool:
-        parent = os.path.join(current_dir, os.path.pardir)
-        return os.path.samefile(current_dir, parent) or any(
-            os.path.isdir(os.path.join(current_dir, cvs_root)) for cvs_root in (".git", ".hg")
-        )
-
-    # Preserve the original behavior, returning PYPROJECT_FILE if exists
-    if os.path.isfile(PYPROJECT_FILE) or is_root(os.path.curdir):
-        return PYPROJECT_FILE
-
-    # And iterate over the tree
-    current_dir = os.path.pardir
-    while not is_root(current_dir):
-        config_file = os.path.join(current_dir, PYPROJECT_FILE)
-        if os.path.isfile(config_file):
-            return config_file
-        parent = os.path.join(current_dir, os.path.pardir)
-        current_dir = parent
-
-    return PYPROJECT_FILE
-
+CONFIG_NAMES: Final = ["mypy.ini", ".mypy.ini"]
+SHARED_CONFIG_NAMES: Final = ["pyproject.toml", "setup.cfg"]
 
-CACHE_DIR: Final = ".mypy_cache"
-CONFIG_FILE: Final = ["mypy.ini", ".mypy.ini"]
-PYPROJECT_FILE: Final = "pyproject.toml"
-PYPROJECT_CONFIG_FILES: Final = [find_pyproject()]
-SHARED_CONFIG_FILES: Final = ["setup.cfg"]
 USER_CONFIG_FILES: Final = ["~/.config/mypy/config", "~/.mypy.ini"]
 if os.environ.get("XDG_CONFIG_HOME"):
     USER_CONFIG_FILES.insert(0, os.path.join(os.environ["XDG_CONFIG_HOME"], "mypy/config"))
 
-CONFIG_FILES: Final = (
-    CONFIG_FILE + PYPROJECT_CONFIG_FILES + SHARED_CONFIG_FILES + USER_CONFIG_FILES
-)
-
 # This must include all reporters defined in mypy.report. This is defined here
 # to make reporter names available without importing mypy.report -- this speeds
 # up startup.

Diff for: mypy/main.py

@@ -564,7 +564,7 @@ def add_invertible_flag(
         "--config-file",
         help=(
             f"Configuration file, must have a [mypy] section "
-            f"(defaults to {', '.join(defaults.CONFIG_FILES)})"
+            f"(defaults to {', '.join(defaults.CONFIG_NAMES + defaults.SHARED_CONFIG_NAMES)})"
         ),
     )
     add_invertible_flag(