-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🚀 Feature: Re-run a test multiple times: this.repeat
#2332
Comments
Hmm. What's the problem with putting the loop inside your test? |
It's just not as convenient - I have to manually call Compare this: (can be improved) describe('#multiple-runs', function() {
var beforeEachFunction = function() {
// Code
};
beforeEach(beforeEachFunction);
it('runs 15 times', function(done) {
var recursive_test = function(n) {
if (n > 15) {
done();
return;
}
beforeEachFunction();
// Code
recursive_test(n + 1);
}
recursive_test(1);
});
}); With this: describe('#multiple-runs', function() {
beforeEach(function(){
// Code
});
it('runs 15 times', function(done) {
this.rerun(15);
// Code
});
}); |
i think a better sintax could be something like this
|
did this function realize ? |
@fantasyRqg no. |
No. But in case someone wants to mess with it, here's some code that does testing manually. Save the following as "multirun.spec.js". var debuglog = require('util').debuglog('multirun');
describe('#multiple-runs', function () {
var NRUNS = 5;
// Returns vector of prime numbers (up to max) [Sieve of Eratosthenes]
function getPrimes(max) {
var primes = [], sieve = [];
for (var i = 2; i <= max; ++i) {
if (!sieve[i]) {
// i has not been marked -- it is prime
primes.push(i);
for (var j = i << 1; j <= max; j += i) {
sieve[j] = true;
}
}
}
return primes;
}
// Returns random integer (up to max)
function getRandomInt(max) {
return Math.floor(Math.random() * Math.floor(max));
}
// Randomize array element order in-place. [Durstenfeld]
function shuffle(array) {
for (var i = array.length - 1; i > 0; i--) {
// Pick a remaining element
var j = Math.floor(Math.random() * (i + 1));
// Swap it with current element
var temp = array[i];
array[i] = array[j];
array[j] = temp;
}
return;
}
//=========================================================
it('should succeed ' + NRUNS + ' times', function (done) {
(function _suite(nruns) {
var candidate, primes, somePrimes;
function _suiteSetup() {
var MAXPRIME = 31;
primes = getPrimes(MAXPRIME);
debuglog('primes[' + primes.length + '] =', primes, '(orig)');
// :NOTE: Remove '2' from list to improve odds of test success
primes = primes.slice(1);
}
function _setup() {
var MAX = 100;
candidate = getRandomInt(MAX);
// (Re)shuffle primes
shuffle(primes);
debuglog('primes[' + primes.length + '] =', primes, '\n');
// Use first NKEEP prime #s for test
var NKEEP = 3;
somePrimes = primes.slice(0, NKEEP);
debuglog('somePrimes[' + somePrimes.length + '] =', somePrimes);
}
// Code being tested [throw to indicate failure]
function _test(run) {
somePrimes.forEach(function (prime) {
debuglog('candidate:', candidate + ', prime:', prime);
if (candidate % prime === 0) {
throw new Error('' + candidate + ' is divisible by ' + prime);
}
});
}
function _teardown() {
candidate = undefined;
somePrimes = undefined;
}
try {
_suiteSetup();
for (var n = 0; n < nruns; n++) {
debuglog('==========');
_setup();
_test(n);
_teardown();
}
}
catch (err) {
return done(new Error('run ' + (n+1) + ' failed: ' + err.message));
}
return done();
})(NRUNS);
});
}); Execute it, expecting both failure/success: $ mocha multirun.spec.js Same as above, with verbose output. $ NODE_DEBUG=multirun mocha multirun.spec.js |
This is also useful for testing that the first run didn't break any future usages of whatever your testing without requiring a restart. |
I am facing the similar issues with tests as well cant run the same test multiple times |
one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness |
Any date prognosis of realization? |
This is what I'm facing. I'll just wrap it in a shell script and run it a couple hundred times. |
Does anyone know if there is any reason why this hasn't been implemented besides no one actually doing it? My use case is testing for leaks in C++ modules. |
Refs: Re-run a test multiple times mochajs#2332
We will see if we will be able to get this into mocha. |
I like cc @mochajs/maintenance-crew - would like to hear from another maintainer before deciding anything. |
this.repeat
@JoshuaKGoldberg In case you are wondering, judicial corruption come first before any Github PRs. |
What does judicial corporation have to do with this? |
I have been living in a total isolation for the last 4 years since an employer covered up a huge judicial scandal involving sex and corruption. I have been working on lots of opensource projects during this time while living on social welfare. The situation has been worsening ever since, with many of my PRs initially being blocked - after which I am signaled by simultaneous comments that if I was to shut up about the affair, they will be willing to merge them. I am also offered a job - which includes a condition of a sexual nature - if I accept to not talk about the affair anymore. @JoshuaKGoldberg's employer is one of the companies involved and he posted this simultaneously with a few other pings. PS. Just as your comment, too. |
Very sorry to hear about that @mmomtchev. @JoshuaKGoldberg himself is an independent freelancer and has no employer. And we're all here to make Mocha the best possible project. Judicial corruption and similar is out of scope of this issue and while that does not at all diminish the importance of such issues, they are better dealt with in other channels 🙏 |
@voxpelli it is not, since I was clearly shown that these two subjects were in fact related. It is a condition that I have been asked in a significant number of open-source projects on Github during the last few years and my only answer can be no. Should anyone have any interest in pushing this PR forward, I am willing to contribute. Should there by any other conditions, my answer is negative. |
I'm not a maintainer but |
Talked with @voxpelli: we think this is a good idea and worth having in. Marking as accepting PRs (i.e. #5011). ✅ Note that per #5027 our main focus is on maintenance rather than features. This particular feature goes a bit against our maintenance goals... but we think it's a straightforward + useful enough edition to justify it. ...and it doesn't hurt that #5011 is a wonderfully written PR (docs! tests! 😍!). |
That's great, thank you. |
Marking as |
this.retries()
allows to run a test multiple times, but the test will pass on the first successful run.I'm suggesting to add
this.rerun()
for tests that might randomly succeed (false positives). It will run the testn
times and will pass only if all of the reruns have passed successfully. It's useful when testing behavior that is probabilistic.At the moment I'm running the test in a loop:
The problem with this approach is that it outputs 100 different tests and messes up the test counting.
The text was updated successfully, but these errors were encountered: