pytest hangs indefinitely after completing tests in parallel #60
Comments
|
does it happen on linux as well? |
|
It seems there's a thread running in background preventing the nodes created by xdist to terminate properly... Also I notice you are trying to forcefully kill the AppEngine process you started... isn't there a cleaner way to do the same, using an API perhaps? |
|
Also, I would suggest taking a look at pytest-beds: Fixtures for testing Google Appengine (GAE) apps. |
|
According to the App Engine docs, "To stop the local server: with Mac OS X or Unix, press Control-C or with Windows, press Control-Break in your command prompt window." (https://cloud.google.com/appengine/docs/python/tools/using-local-server). Modifying the above code to send the Control-Break signal: Now actually terminates pytest: This occurs whether xdist is used or not. P.S. It looks like pytest-beds only has fixtures for the testbed, which is not used for end to end tests that require the actual server. Also, I make my own fixtures for the actual tests. The above code is just for demonstration and xdist testing. P.P.S. I will take a look at Linux when I get a change to setup App Engine on my Linux machine. I suspect the --boxed option may fix this, but is not available on windows. |
|
the |
|
@PaintingInAir Sorry, it wasn't clear to me: did send the signal using |
|
@nicoddemus There appears to be a workaround for the Regardless, it appears the underlying issue is a loose thread that causes xdist to hang. I guess the first question is whether or not xdist should be able to handle this situation. Note that pytest can finish under these circumstances, when xdist is not used. If this is a bug worth fixing, it is probably worth isolating the underlying cause (such as a loose thread) first. |
|
This is happening on AppVeyor. |
|
I've experienced this as well and had to switch to Also: I noticed the issue less when I was using |
|
also having this problem, does anyone fixed it? |
|
I am seeing the same issue on linux. Is there any fix/workaround? I am using pytest version 3.1.0. |
|
Hi, |
|
Hi, Unfortunately it is hard to nail down the problem; usually it is not a bug in pytest-xdist per-se, but some bad interaction of the system under testing. Is anybody able to provide a minimal reproducible example? |
|
Seeing same issue while running in a Docker container; image based off of here is gist with minimal reproducible example: https://gist.github.com/rohansb/3699278d4f7388c3881ced81a0fa66c9 |
|
@rohansb thanks, but strangely I don't even see |
|
@nicoddemus that's right, does that indicate its not related to .. this is where it hangs indefinitely, off-course! |
|
FYI.. issue persists even after installing |
|
Just thinking -- It may be infeasible to solve for all envs all the time, it's actually a somewhat hard problem Something that could be tried: if one cannot put capture=no overall, could tell it to capture output and logs to file and never stdout/stderr. If needed, you could use pytest hookspec to cat those tempfiles at end of session as needed |
|
I'm having this same problem in a different context. I'm running pytest-django tests locally with xdist, and it hangs in the same way. Based on the conversation here, I tried running them with the --capture=no option, but still had the problem. Any suggestions on how to debug & work around this in my case? |
|
I would try to attach a remote debugger and see where the processes are getting stuck (PyCharm has a remote debugger for example). |
|
Had the same issue running pytest with xdist in a docker. Docker never terminted after test completion. Pytest on pid 1 was unkillable and no stop signal could kill it. --boxed fixed the issues but broke scopes for fixtures resulting in a lot of setup and tear down. --dist loadscope fixed the issue but had (acceptable in my case) impact on how the cli received log output. I pass it to reportportal anyways. ¯_(ツ)_/¯ |
|
I've dumped stack traces (with Pyrasyte) from the hanged master process. It has many threads like this: Most probably, these threads communicate with spawned subprocesses. And there is one thread, which hangs on the Queue: |
|
I am having same issue, tests works fine when tried to debug using debugger in vscode. It seems there is some problem with threads. |
|
hi, i experienced the same problem with my |
|
If it can help someone, I had the same symptom with pytest hanging The root cause was that postgresql requires to close all sessions, otherwise there are still open locks |
|
🔥 Solved here (SQLAlchemy Session and Sessionmaker + Pytest Fixture + Coverage interaction) 🔥 Here's the backstory: The host system is a FastAPI application and we started to have the same issue with our test suite when using We have a generalized way of grabbing a session_factory: Callable[[], Session] = sessionmaker(
autocommit=False, autoflush=True, bind=dors_engine
)
def get_session() -> Iterator[Session]:
db_session: Session = session_factory()
try:
yield db_session
finally:
db_session.close()And this is the test fixture we had: @pytest.fixture
def db_session(app_config) -> Session:
yield from get_session()The We changed the fixture to include a new-ish SQLAlchemy 1.4 utility function (that says in its docstring that "might be useful for test scenarios" - so I guess there's something to that) that closes all outstanding sessions it has a weakref to. @pytest.fixture
def db_session(app_config) -> Session:
yield from get_session()
##########################
# This needs to be here because tests hang when run with "coverage" or "pytest --cov"
# The problem does not arise when using a simple `pytest` invocation
# Something in the way coverage's hooks work might be allowing database sessions
# to become dangling/lost in context (and never freed)
##########################
from sqlalchemy.orm import close_all_sessions
close_all_sessions()And just like that, our test suite was back up and running, both locally and on GitHub Actions (No need to downgrade or play around with pytest / coverage versions or anything like that). Hope this helps someone else! |
|
Just spent an unpleasant couple of weeks trying to find the reason for this in our codebase. Thanks @flipbit03 and @gpouilloux for the hint. SQLA sessions are stored as weak refs (see code here) and we use pypy which is a bit lazy about garbage-collection. I assume (but may be wrong) that pytest coverage can also interfering with the gc mechanism for these sessions too. What that means practically is that if you accidentally check out a session in a fixture and forget to remove it and there is no gc cycle you can get a locked table when you try to drop it. Either a |
I run into this behavior when running web tests with Google App Engine. However, since this issue only occurs when using xdist, I do not believe it is caused by App Engine itself.
Run the following two tests in parallel. Tests will complete, but pytest will not finish:
I ran the tests with the following line:
py.test -k REMOVE -n 2The result is:
Killing the process results in the following traceback:
Finally, I am on windows.
Note that this test is run with the hello world application given by Google. Any other app engine application I've tested causes the same hang.
Also, I have tried a Popen that simply runs python, and this does not result in the same hang.
The text was updated successfully, but these errors were encountered: