Add support for stress tests to nextest #2219
Replies: 7 comments
-
Have done quite a bit of this in the past with Go. Some notes that may help: In Go, tests are fairly similar; there are a few test binaries that get compiled then you can execute them like We use a tool https://github.com/howardjohn/golang-tools/tree/master/cmd/stress - there is an official Go variant but the fork has a lot more features. Running it will output something like Typically this is used on a single test or test package, focusing on a specific area, but we also use a wrapper that allows running on ~all tests, with some fixed test count or duration. This allows doing something like "run every test for 1000 executions or 1 min (whichever is fastest)" to analyze the entire codebase for flaky tests. So overall, I think nice features are:
In the meantime, I also created a bash helper that does this. Its a bit overcomplex and poor UX, but usage is like function rust-flaker() {
relpath () {
python -c "import os.path; print(os.path.relpath('$1','${2:-$PWD}'))"
}
clr='\e[0m'
blue='\e[0;34m'
executables="$(cargo test --no-run --message-format json | jq -R 'fromjson? | select(type == "object") | select(.executable != null) | select(.profile.test == true) | select(.target.kind | contains(["bin"]) == false)| .executable' -r)"
want_name="${1}"
local bin=""
while read -r line; do
name="$(basename $line | cut -d- -f1)"
if [[ "${want_name}" == "${name}" ]]; then
echo "${blue}Found test binary: $(relpath $line)${clr}"
bin=${line}
break
fi
done <<< "${executables}"
if [[ "${bin}" == "" ]]; then
"Test binary '${want_name}' not found, have:"
while read -r line; do
echo " $(basename $line | cut -d- -f1)"
done <<< "${executables}"
return 1
fi
echo "${blue}Running test once...${clr}"
$bin "${2:-}" || return 1
echo "${blue}Running test repeatedly...${clr}"
filter="${2:-}"
shift; shift
stress "$@" $bin "${filter}"
} |
Beta Was this translation helpful? Give feedback.
-
Thanks for the detailed overview @howardjohn! |
Beta Was this translation helpful? Give feedback.
-
Hi, @sunshowers Is this feature supported now? |
Beta Was this translation helpful? Give feedback.
-
Hi @iamazy. No, the feature isn't yet supported. I don't expect to get to it any time at least in 2023 either, I'm too busy with other work. However, I'd be interested in both design and engineering contributions towards this. I think the list of concerns mentioned in @howardjohn's post is a good start. I know @bmwill expressed some interest in implementing it but I believe he's been super busy as well. |
Beta Was this translation helpful? Give feedback.
-
Would love to see this implemented |
Beta Was this translation helpful? Give feedback.
-
Adding my two cents on the proposed design. Would like to see parallelization and repeats as separate configs. Or ensure the |
Beta Was this translation helpful? Give feedback.
-
The need for stress testing at my workplace keeps going up, so I'm hoping to carve out some time for this within the next 3 months. |
Beta Was this translation helpful? Give feedback.
-
Add support to run a single test multiple times in parallel. This is a somewhat different mode of operation from nextest as usual (the runner should be fed the same test multiple times, and should maybe not run other tests at the same time), but is worth doing as a future improvement.
Beta Was this translation helpful? Give feedback.
All reactions