Ignore patterns not passed to watchfiles, leading to “file watch limit reached”

[scy]

This project I’m working on uses a shared repository for a FastAPI backend and a JavaScript frontend, with the frontend being a subdirectory of the backend. Thus, the file structure looks something like this:

• /home/scy/proj/myproject/
  • myproject/ (package)
    • [Python files]
  • myproject-frontend/
    • [JavaScript files]
  • pyproject.toml
  • setup.cfg
  • .venv/
  • […]

As you can see, the Python files are in the myproject package, but the top-level Python “project” directory contains the frontend. (Yes, this probably isn’t optimal, but that’s where I’m at right now.)

This means that, by default, running Uvicorn with --reload will try to watch the whole frontend folder, including all of the thousands of items in node_modules. This results in an error message like FileNotFoundError: OS file watch limit reached. about ["/home/scy/proj/myproject/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources"].

If I delete node_modules, the error goes away.

Now, of course I could just increase the file watching limit, but it’s completely unnecessary for Uvicorn to watch the frontend anyway. The first thing I tried was adding node_modules to --reload-excludes. Turns out that this has no effect.

First of all, Uvicorn will handle everything that it can’t resolve to a directory name as a pattern, which means that it will filter files named node_modules. I could try crafting a pattern that recursively matches files inside of a directory named that way, something like **/node_modules/**, but this wouldn’t help either: Uvicorn doesn’t pass the filters to watchfiles at all! Instead, it instructs watchfiles to simply watch everything in --reload-dir and then discards every change notification that isn’t relevant.

So, with the way that it’s implemented right now, the only way to reduce the number of files watched is to be selective about --reload-dir, because --reload-exclude doesn’t help.

Sure, I can do that (and I will), but it isn’t at all clear from the documentation that

1. --reload-exclude expects you to provide a file pattern; it doesn’t work with directory names
2. using --reload-exclude will not save resources at all.

Is there a way to exclude directories with --reload-exclude at all?

[scy]

Okay, turns out that even passing --reload-dir myproject (i.e. the package directory) won’t help, because Uvicorn will always add the current working directory to the list of watched directories, which in my case needs to be the repo root, which also includes the subdirectory with the JS frontend.

The only way to solve this seems to be changing the repo structure to completely separate Python & JS into different parent directories, without one being a parent of the other, and start Uvicorn with the CWD set to the Python directory instead of the repo root. And sure, you might argue that this is the cleanest way to structure the repo anyway, but I think that requiring the CWD to be watched can be problematic in other cases, too.

Also, it means that Uvicorn is lying about which directories it’s watching, because this list doesn’t account for the CWD special case.

[Oduig]

My two cents, here is a monkey patch that can circumvent the issue. Insert this in your code before you start uvicorn.

        import watchfiles
        reload_dirs = [...] # Fill this with pathlib.Path
        # Monkey patch to fix infinite watch loop when watching log files.
        # See https://github.com/encode/uvicorn/discussions/1978
        original_watch = watchfiles.watch
        def patched_watch(*_, **kwargs): # Ignore CWD path that uvicorn passes as args to watchfiles.watch
            return original_watch(*reload_dirs, **kwargs)
        watchfiles.watch = patched_watch
        cnx_app.run('auth_service.app:create_app', reload_dirs=reload_dirs)

As @scy mentioned, Uvicorn always adds the current working directory to watchfiles. This patch makes it so that the arguments from Uvicorn are discarded, and it only uses reload_dirs instead to watch files.