How can I use pytest within an embedded Python environment with __debug__ == False?

[bradleyharden]

I'm trying to use pytest within an application that uses Python as an embedded scripting language. However, it looks like the application invokes Python with the equivalent of -O, and __debug__ is False. Thus, none of my asserts actually raise an AssertionError, and much of pytest is rendered useless.

What options do I have here? I suppose I could manually raise AssertionErrors, but that gives up a lot of the benefits of assertion rewriting. It would at least give me a proper unit test framework, but it's certainly less than ideal. Are there other options?

[bradleyharden]

As I've been looking into this, I'm confused why pytest doesn't work here. According to this blog post and a quick skim of the code, it seems like assertion rewriting should automatically convert all existing assert statements to a raise AssertionError equivalent. That's pretty much exactly what I need. Yet it doesn't work? Why?

[bradleyharden]

After some digging, it seems like my tests are not getting rewritten because of a strange type mismatch in AssertionRewritingHook.find_spec.

pytest/src/_pytest/assertion/rewrite.py

Lines 103 to 115 in ac41898

	spec = self._find_spec(name, path) # type: ignore
	if (
	# the import machinery could not find a file to import
	spec is None
	# this is a namespace package (without `__init__.py`)
	# there's nothing to rewrite there
	or spec.origin is None
	# we can only rewrite source files
	or not isinstance(spec.loader, importlib.machinery.SourceFileLoader)
	# if the file doesn't exist, we can't rewrite it
	or not os.path.exists(spec.origin)
	):
	return None

Specifically, the class of spec.loader does not seem to match importlib.machinery.SourceFileLoader. The following code

print(f"{spec.loader.__class__}")
print(f"{importlib.machinery.SourceFileLoader}")
print(isinstance(spec.loader, importlib.machinery.SourceFileLoader))

yields

<class '_frozen_importlib_external.SourceFileLoader'>
<class '_frozen_importlib_external.SourceFileLoader'>
False

If I remove the isinstance check, my assertions are rewritten and everything works. Do you have any idea what might be going on?

[bradleyharden]

Interestingly,

print(isinstance(spec.loader, importlib.abc.FileLoader))
print(isinstance(spec.loader, importlib.abc.SourceLoader))

yields

False
False

This really doesn't make any sense to me. Are there somehow two separate instances of importlib coexisting?

[bradleyharden]

After more digging, I can see some limited evidence that the application has modified parts of importlib. My current theory is that the various find_spec functions are returning an instance of a SourceFileLoader class that has since been replaced within importlib and is no longer reachable by import.

At this point, I'm pretty confident there's nothing any of you can do to help me. For the moment, my workaround is to modify isinstance(spec.loader, importlib.machinery.SourceFileLoader to be spec.loader.__class__.__name__ == "SourceFileLoader". If anyone has a better idea, I would love to hear it.

[bradleyharden]

It looks like sys.path_hooks[1] returns instances of an old FileFinder class that is not reachable at importlib.machinery.FileFinder. The spec objects returned by its find_spec method also refer to an old version of SourceFileLoader. That's what causes the isinstance check to fail.

The simplest solution I've found is to extract a reference to the old class and replace the new class, i.e.

importlib.machinery.SourceFileLoader = sys.path_hooks[1].__globals__["SourceFileLoader"]

As best I can tell, this doesn't cause any side effects.

This has been a strange adventure, but I think it ends here.

[RonnyPfannschmidt]

Thanks for providing the follow up

For future sanity it May be helpful to work with your vendor so they can provide a better pytest integration out of the box

[bradleyharden]

Yeah, that would have been a lot easier, but for a number of reasons it probably wouldn't have happened. I'm trying to replace our existing system for tests, which essentially amounts to a very primitive unit testing framework. I can't really get buy-in for that without showing people how it could be better. And buy-in would be a prerequisite to requesting vendor support.

[RonnyPfannschmidt]

Having the vendor in the Loop might be a help

Selling a duct-tape pytest integrate to a higher up May be harder than selling a update from the vendor

But that might be undue optimism on my side