Do not show logs for failed tests by default

[lorenyu]

Ticket

Changes

• Turn off showing of captured logs by default in failed tests

Context for reviewers

Logs can be useful but are generally noisy to include by default for failed tests so this PR turns it off by default.
Turning them on can be done with make test args="--show-capture=all"

Testing

Example now by default

[chouinar]

Might be good to have a quick sync chat on this. While I do want to keep my setup working, I don't want to dictate the implementation so much so for just myself.

Ideally the default would be roughly the same verbosity as it was before the changes to avoid drowing out the test output, but configurable enough for when you do want the full logs.

I did like your idea about a separate .env file for an individual developer, but what would be the best way to make it clear that exists?

[lorenyu]

Sure happy to chat about it. I'll look for some time.

If I understand correctly, before the changes either the console handler we defined would be attached to the root logger, or the caplog handler would be attached (if the developer included the logging_fix fixture which turned off the console handler), but not both.

the console handler verbosity can be controlled via LOG_LEVEL, which could either be in local.env or a separate test.env file if we go that route. the caplog verbosity can be controlled by caplog.setLevel(.

also by default, neither the logs that get printed to stdout stream from console handler nor the logs captured by caplog get shown to the user unless a test fails, and at that point both sets of logs get printed out.

so maybe we can chat about what the ideal would be in each / all of these scenarios, and how it could potentially be flexible to individual developer preferences.

[chouinar]

I think I've got an idea, not sure the best way to configure it, but rather than messing with log levels, what about the stream handler itself?

Right now, we configure the stream handler in logging/config.py as console_handler = logging.StreamHandler(sys.stdout), but what if that was overridable in some way?

If we make a real simple stream ourselves:

class NullStream(StringIO):
    def write(self, text:str) -> None:
        pass

and then do logging.StreamHandler(NullStream()) - we get no logs.

I'm not sure how we would want to configure this though. I think this might be best handled by the user-specific env file we've thought about, unless you've got another idea?

[lorenyu]

That seems reasonable. But it seems like it has the same effect as changing the log level while requiring more code?

I'm imagining it'd be the difference in local.env of log_handler=console vs log_level=CRITICAL or log_level=WARNING

Do you think the question boils down to how we can have user-specific env files? If so, I think we can rename local.env to .env, remove it from source control, provide an example.env, and add something along the lines of cp local.env .env if .env doesn't already exist to make setup-local. i think we can do that using Makefile's built in method of detecting whether a target already exists i.e.

.env:
  cp example.env .env

setup-local: .env
  ...