feat: add option to use posix exit code upon fatal signal

[73rhodes]

Requirements

• Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
• All new code requires tests to ensure against regressions.

Description of the Change

Mocha uses the number of failed tests as an exit code, which is unconventional and has led to a number of issues when trying to integrate mocha into ci/cd pipelines.

To resolve these issues, this PR introduces a --posix-exit-codes boolean command-line option. If this option is specified, a fatal signal (eg. SIGABRT, et al) will cause the process to consistently exit with a standard posix exit code (128 + the numeric ID of the signal) when mocha is run as a child process (which is the case when passing node options). This helps to solve issues for toolchains that expect standard posix exit codes, for example by preventing out-of-memory crashes from being silently ignored.

The GNU libc manual page provides additional context on why using the number of errors as an exit status is problematic:

Warning: Don’t try to use the number of errors as the exit status. This is actually not very useful; a parent process would generally not care how many errors occurred. Worse than that, it does not work, because the status value is truncated to eight bits. Thus, if the program tried to report 256 errors, the parent would receive a report of 0 errors—that is, success. For the same reason, it does not work to use the value of errno as the exit status—these can exceed 255.

Providing the option to exit with standard posix shell exit codes avoids these problems and solves a number of downstream issues listed below.

Alternate Designs

The alternatives considered were not in the scope of the mocha project.

Why should this be in core?

This option is implemented in the core repository where signal handling occurs.

Benefits

This PR provides a solution for #3559 and various downstream issues reported by mocha consumers; it preserves non-zero exit codes when mocha is spawned as a child process via various CI/CD or reporting tools and prevents silently swallowing out-of-memory errors.

Possible Drawbacks

Introduces another option. Requires docs. Leaves non-standard exit code behavior to remain as the default.

Applicable issues

fixes #3559
#3893
#2445
#2438
istanbuljs/nyc#798
cypress-io/cypress#24695

• No breaking changes; this is an enhancement (minor release)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: 73rhodes / name: Darren DeRidder (8999b7b, ad30057, 3b99517, b6e6d30, b71eaf3, 0fb6ca3, 8a4f2bd, 817d890, 62528c5, 34f2822, 59e471d, 2150c30, d2991a1, 7cf79d8, ca78dbc, 609f94f, 26bebac, 0aa939f, 1809576, 592f071, 9ee844a, 6d237fc, 3438da6, df38431)

[github-actions]

This PR hasn't had any recent activity, and I'm labeling it stale. Remove the label or comment or this PR will be closed in 14 days. Thanks for contributing to Mocha!

[JoshuaKGoldberg]

We should probably disable the stale action, pending picking up reviewing PRs.

[JoshuaKGoldberg]

Nice and clean implementation, thanks! ✨

Requesting changes on a few things. But let me know, please, if I'm off base here 🙂.

[JoshuaKGoldberg]

Note that after this lands, we'll want to file a followup issue about making this the default behavior in some future version of Mocha.

[73rhodes]

@JoshuaKGoldberg Thanks for the review! Hopefully that latest update addresses your questions, but let me know if there's anything else you'd like to see. Looking froward to not using our own fork of mocha! 😆

[JoshuaKGoldberg]

Progress! Thanks for adding in the tests. I think we'll need to capture some more cases?

[JoshuaKGoldberg]

Will take a look, thanks!

[JoshuaKGoldberg]

This looks wonderful to me, thanks so much for all the discussion and iteration @73rhodes! 🙌

Marking this as having one team member approval. We'll want another team member to take a look too. Ping @voxpelli, IIRC you were particularly interested in making Mocha more standards-compliant?

[coveralls]

coverage: 94.371% (+0.01%) from 94.358%
when pulling 59e471d on zendesk:master
into 99601da on mochajs:master.

[73rhodes]

@JoshuaKGoldberg just circling back to see if there's anything you need from me to unblock this enhancement? lmk 🙏

[JoshuaKGoldberg]

Thanks for the ping! Nothing yet. The other maintainers have been swamped (so much happening in the ecosystem!) but I'm still holding onto hope someone will be able to review. I've personally time boxed this one to within June.

[voxpelli]

As far as I can see this only changes the flow in this flow:

mocha/bin/mocha.js

Line 80 in 2f3fedc

if (mochaArgs['node-option'] || Object.keys(nodeArgs).length || hasInspect) {

Not in the other flow:

mocha/bin/mocha.js

Lines 139 to 141 in 2f3fedc

	} else {
	debug('running Mocha in-process');
	require('../lib/cli/cli').main([], mochaArgs);

Is this really the right place for this fix?

Eg: The non-standard error codes are set here:

mocha/lib/cli/run-helpers.js

Lines 26 to 45 in 2f3fedc

	const exitMochaLater = code => {
	process.on('exit', () => {
	process.exitCode = Math.min(code, 255);
	});
	};
	/**
	* Exits Mocha when Mocha itself has finished execution, regardless of
	* what the tests or code under test is doing.
	* @param {number} code - Exit code; typically # of failures
	* @ignore
	* @private
	*/
	const exitMocha = code => {
	const clampedCode = Math.min(code, 255);
	let draining = 0;
	// Eagerly set the process's exit code in case stream.write doesn't
	// execute its callback before the process terminates.
	process.exitCode = clampedCode;

And in the old-school case where one forces the process to be killed once all tests are done it happens within that:

mocha/lib/cli/run-helpers.js

Line 52 in 2f3fedc

process.exit(clampedCode);

---

I overall find this PR to be a mix of two changes:

1. Stop mocha from abusing the exit code as an error count reporter
2. Add signal specific exit codes

The latter of the two I would want to read up on more, but the first of them should be an easier fix.

[voxpelli]

Node.js should already be setting the signal exit codes, I wonder if we can piggy back on that instead? https://nodejs.org/api/process.html#exit-codes

[73rhodes]

@voxpelli Thanks for the review. That's right, the option only applies "when mocha is run as a child process". The reasoning was to try and be as unobtrusive as possible and address specifically the problem of fatal errors (OOM et al) getting swallowed silently in a CI/CD pipeline where mocha is spawned by something like nyc. It sounds like there's general agreement that mocha shouldn't use the exit code to report the number of errors though, so I can update this PR to support the option in both cases. As for exit codes other than 0, 1 and 128+signal - I guess we'd need to know what the exit code scheme should be and could be something for another PR....

[voxpelli]

@73rhodes Is there a need to handle 128+signal codes manually or is that already handled by node.js itself? Reason why I ask is since its typically not something I have seen handled manually by other CLI tools built with node.js

[73rhodes]

@voxpelli ... I just updated the PR with a commit (this PR) that adds the signal-handling behavior for both the child-process and in-process modes of running mocha, as requested. As far as piggybacking on node's signal handling, I'm just working with these inputs:

proc.on('exit', (code, signal) => {

and

const exitMochaLater = code => {

where, based on what I've seen and implemented in the tests, we need to check the code and signal and decide on the desired exit code ourselves. Maybe it's possible that at some deeper layer of mocha internals there's a different way to implement this, but this has been working solidly in production.