Halts when waiting for a browser to exit after TestLoader Failure

[omairvaiyani]

We are using ember exam with the following flags--split=8 --parallel. Our CI pipeline was running for multiple hours because the ember exam step did not exit.

In the logs we noticed that one of the browsers exited:

Test Failure - Exam Partition 7 - TestLoader Failures: <some-test-file>: could not be loaded
	 Died on test #1     at Object.test (webpack://__ember_auto_import__/../../common/temp/node_modules/.pnpm/[email protected]/node_modules/qunit/qunit/qunit.js?:464:34): Could not find module `<some-source-file>` imported from `<some-test-file>` 

However after this exit when the remaining partitions completed, this was logged:

.....Out of requested 8 browser(s), 7 browser(s) was launched & completed.
Waiting for remaining browsers to exited.

The process does not exit, ever. Regardless of the bug, I couldn't find any flags in the docs that would allow a max wait period to avoid such a situation?

[tniezurawski]

I'm unaware of such a flag and have a feeling like it might be connected more to testem than ember-exam, but that's just a guess 🤷‍♂️

I can only advise using built-in timeouts in CI services. CircleCI has a nice no_output_timeout flag which is low by default - 10 minutes without output and it terminates the job. GitHub actions' timeout on the other hand is designed to get money from you 😅 The default limit is high - 6 hours. So the first thing for GH Actions that I do is to set a low timeout 😉

[acorncom]

I ran into a similar variant of this issue just recently for a client on an app. If we run a ember exam --load-balance --parallel=4 command and have an uncaught error occurring in the test app (in my case, a Google Maps RefererNotAllowedMapError was being thrown), then it is causing ember-exam to lock up and never exit out with errors (CircleCI would kill the tests after 10 minutes).

This started happening in our app after an upgrade to ember-exam 9 (from 5) and testem 3.15.2 (from 3.10.0). Unsure if I'll have time to put a reproduction together later, but wanted to note the issue we were seeing in case it helps us sort out what is at the heart of things.

Running the same set of tests without load-balancing mode works without issue (with or without randomization), so it seems that load-balanced mode is slightly more sensitive to errors/issues that occur within the test suite