nextest-rs · benesch · Dec 26, 2024 · Jan 1, 2025 · Jan 1, 2025 · Jan 2, 2025
diff --git a/site/mkdocs.yml b/site/mkdocs.yml
@@ -112,7 +112,10 @@ nav:
                 - "Mutual exclusion and rate-limiting": docs/configuration/test-groups.md
                 - "Environment variables": docs/configuration/env-vars.md
                 - "Extra arguments": docs/configuration/extra-args.md
-                - docs/configuration/setup-scripts.md
+          - "Scripts":
+                - "Overview": docs/scripts/index.md
+                - docs/scripts/setup.md
+                - docs/scripts/pre-timeout.md
     - Machine-readable output:
           - "About output formats": docs/machine-readable/index.md
           - "JUnit support": docs/machine-readable/junit.md

diff --git a/site/src/docs/scripts/index.md b/site/src/docs/scripts/index.md
@@ -0,0 +1,125 @@
+---
+icon: material/script-text
+---
+
+# Scripts
+
+Nextest supports running _scripts_ when certain events occur during a test run. Scripts can be scoped to particular tests via [filtersets](../filtersets/index.md).
+
+Nextest currently recognizes two types of scripts:
+
+* [_Setup scripts_](setup.md), which execute at the start of a test run.
+* [_Pre-timeout scripts_](pre-timeout.md), which execute before nextest terminates a test that has exceeded its timeout.
+
+Scripts are configured in two parts: _defining scripts_, and _specifying rules_ for when they should be executed.
+
+## Defining scripts
+
+Scripts are defined using the top-level `script.<type>` configuration.
+
+For example, to define a setup script named "my-script", which runs `my-script.sh`:
+
+```toml title="Setup script definition in <code>.config/nextest.toml</code>"
+[script.setup.my-script]
+command = 'my-script.sh'
+# Additional options...
+```
+
+See [_Defining setup scripts_](setup.md#defining-setup-scripts) for the additional options available for configuring setup scripts.
+
+To instead define a pre-timeout script named "my-script", which runs `my-script.sh`:
+
+```toml title="Pre-timeout script definition in <code>.config/nextest.toml</code>"
+[script.pre-timeout.my-script]
+command = 'my-script.sh'
+# Additional options...
+```
+
+See [_Defining pre-timeout scripts_](pre-timeout.md#defining-pre-timeout-scripts) for the additional options available for configuring pre-timeout scripts.
+
+### Command specification
+
+All script types support the `command` option, which specifies how to invoke the script. Commands can either be specified using Unix shell rules, or as a list of arguments. In the following example, `script1` and `script2` are equivalent.
+
+```toml
+[script.<type>.script1]
+command = 'script.sh -c "Hello, world!"'
+
+[script.<type>.script2]
+command = ['script.sh', '-c', 'Hello, world!']
+```
+
+### Timeouts
+
+All script types support the following timeout options:
+
+- **`slow-timeout`**: Mark a script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
+- **`leak-timeout`**: Mark scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
+
+
+```toml title="Script definition with timeouts"
+[script.<type>.my-script]
+command = 'script.sh'
+slow-timeout = { period = "60s", terminate-after = 2 }
+leak-timeout = "1s"
+```
+
+### Namespacing
+
+Script names must be unique across all script types.
+
+This means that you cannot use the same name for a setup script and a pre-timeout script:
+
+```toml title="Pre-timeout script definition in <code>.config/nextest.toml</code>"
+[script.setup.my-script]
+command = 'setup.sh'
+
+# Reusing the `my-script` name for a pre-timeout script is NOT permitted.
+[script.pre-timeout.my-script]
+command = 'pre-timeout.sh'
+```
+
+## Specifying rules
+
+In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can configure a setup script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
+
+```toml title="Basic rules"
+[[profile.default.scripts]]
+filter = 'rdeps(db-tests)'
+setup = 'db-generate'
+```
+
+(This example uses the `rdeps` [filterset](../filtersets/index.md) predicate.)
+
+Scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
+
+```toml title="Platform-specific rules"
+[[profile.default.scripts]]
+platform = { host = "cfg(unix)" }
+setup = 'script1'
+```
+
+A set of scripts can also be specified. All scripts in the set will be executed.
+
+```toml title="Multiple setup scripts"
+[[profile.default.scripts]]
+filter = 'test(/^script_tests::/)'
+setup = ['script1', 'script2']
+```
+
+Executing pre-timeout scripts follows the same pattern. For example, you can configure a pre-timeout script for every test that contains `slow` in its name.
+
+```toml title="Basic pre-timeout rules"
+[[profile.default.scripts]]
+filter = 'test(slow)'
+pre-timeout = 'capture-backtrace'
+```
+
+A single rule can specify any number of setup scripts and any number of pre-timeout scripts.
+
+```toml title="Combination rules"
+[[profile.default.scripts]]
+filter = 'test(slow)'
+setup = ['setup-1', 'setup-2']
+pre-timeout = ['pre-timeout-1', 'pre-timeout-2']
+```
diff --git a/site/src/docs/scripts/pre-timeout.md b/site/src/docs/scripts/pre-timeout.md
@@ -0,0 +1,69 @@
+---
+icon: material/timer-sand-empty
+status: experimental
+---
+
+<!-- md:version 0.9.59 -->
+
+# Pre-timeout scripts
+
+!!! experimental "Experimental: This feature is not yet stable"
+
+    - **Enable with:** Add `experimental = ["pre-timeout-scripts"]` to `.config/nextest.toml`
+    - **Tracking issue:** [#TODO](https://github.com/nextest-rs/nextest/issues/TODO)
+
+
+Nextest runs *pre-timeout scripts* before terminating a test that has exceeded
+its timeout.
+
+Pre-timeout scripts are useful for automatically collecting backtraces, logs, etc. that can assist in debugging why a test is slow or hung.
+
+## Defining pre-timeout scripts
+
+Pre-timeout scripts are defined using the top-level `script.pre-timeout` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
+
+```toml title="Script definition in <code>.config/nextest.toml</code>"
+[script.pre-timeout.my-script]
+command = 'my-script.sh'
+```
+
+See [_Defining scripts_](index.md#defining-scripts) for options that are common to all scripts.
+
+Pre-timeout scripts do not support additional configuration options.
+
+Notably, pre-timeout scripts always capture stdout and stderr. Support for not capturing stdout and stderr may be added in the future in order to support use cases like interactive debugging of a hung test.
 } else if *will_terminate { 
     let (required_status_level, style) = if retry_data.is_last_attempt() { 
         (StatusLevel::Fail, self.styles.fail) 
     } else { 
         (StatusLevel::Retry, self.styles.retry) 
     }; 
     if retry_data.total_attempts > 1 
         && self.status_levels.status_level > required_status_level 
     { 
         write!( 
             writer, 
             "{:>12} ", 
             format!("TRY {} TRMNTG", retry_data.attempt).style(style) 
         )?; 
     } else { 
         write!(writer, "{:>12} ", "TERMINATING".style(style))?; 
     }; 
 } 
 - with code 0xc00001b2: Invalid access to memory location. (os error 998) 
 /// The current terminating state of a test or script process. 
 /// 
 /// Part of [`UnitState::Terminating`]. 
 #[derive(Clone, Debug)] 
 pub struct UnitTerminatingState { 
     /// The process ID. 
     pub pid: u32, 
     /// The amount of time the unit ran for. 
     pub time_taken: Duration, 
     /// The reason for the termination. 
     pub reason: UnitTerminateReason, 
     /// The method by which the process is being terminated. 
     pub method: UnitTerminateMethod, 
     /// How long has been spent waiting for the process to exit. 
     pub waiting_duration: Duration, 
     /// How much longer nextest will wait until a kill command is sent to the process. 
     pub remaining: Duration, 
 } 
 } else if *will_terminate { 
     let (required_status_level, style) = if retry_data.is_last_attempt() { 
         (StatusLevel::Fail, self.styles.fail) 
     } else { 
         (StatusLevel::Retry, self.styles.retry) 
     }; 
     if retry_data.total_attempts > 1 
         && self.status_levels.status_level > required_status_level 
     { 
         write!( 
             writer, 
             "{:>12} ", 
             format!("TRY {} TRMNTG", retry_data.attempt).style(style) 
         )?; 
     } else { 
         write!(writer, "{:>12} ", "TERMINATING".style(style))?; 
     }; 
 } 
 - with code 0xc00001b2: Invalid access to memory location. (os error 998) 
 /// The current terminating state of a test or script process. 
 /// 
 /// Part of [`UnitState::Terminating`]. 
 #[derive(Clone, Debug)] 
 pub struct UnitTerminatingState { 
     /// The process ID. 
     pub pid: u32, 
  
     /// The amount of time the unit ran for. 
     pub time_taken: Duration, 
  
     /// The reason for the termination. 
     pub reason: UnitTerminateReason, 
  
     /// The method by which the process is being terminated. 
     pub method: UnitTerminateMethod, 
  
     /// How long has been spent waiting for the process to exit. 
     pub waiting_duration: Duration, 
  
     /// How much longer nextest will wait until a kill command is sent to the process. 
     pub remaining: Duration, 
 } 
+
+### Example
+
+To invoke GDB to dump backtraces before a hanging test is terminated:
+
+```toml title="Advanced pre-timeout script definition"
+[script.pre-timeout.gdb-dump]
+command = ['sh', '-c', 'gdb -p $NEXTEST_PRE_TIMEOUT_TEST_PID -batch -ex "thread apply all backtrace"']
+# TODO options
+```
+
+## Specifying pre-timeout script rules
+
+See [_Specifying rules_](index.md#specifying-rules).
+
+## Pre-timeout script execution
+
+A given pre-timeout script _S_ is executed when the current profile has at least one rule where the `platform` predicates match the current execution environment, the script _S_ is listed in `pre-timeout`, and a test matching the `filter` has reached its configured timeout.
+
+Pre-timeout scripts are executed serially, in the order they are defined (_not_ the order they're specified in the rules). If any pre-timeout script exits with a non-zero exit code, an error is logged but the test run continues.
+
+Nextest will proceed with graceful termination of the test only once the pre-timeout script terminates. See [_How nextest terminates tests_](#defining-pre-timeout-scripts). If the pre-timeout script itself is slow, nextest will apply the same termination protocol to the pre-timeout script.
+
+The pre-timeout script is not responsible for terminating the test process, but it is permissible for it to do so.
+
+Nextest executes pre-timeout scripts with the same working directory as the test and sets the following variables in the script's environment:
+
+* **`NEXTEST_PRE_TIMEOUT_TEST_PID`**: the ID of the process running the test.
+* **`NEXTEST_PRE_TIMEOUT_TEST_NAME`**: the name of the running test.
+* **`NEXTEST_PRE_TIMEOUT_TEST_BINARY_ID`**: the ID of the binary in which the test is located.
+* **`NEXTEST_PRE_TIMEOUT_TEST_BINARY_ID_PACKAGE_NAME`**: the package name component of the binary ID.
+* **`NEXTEST_PRE_TIMEOUT_TEST_BINARY_ID_NAME`**: the name component of the binary ID, if known.
+* **`NEXTEST_PRE_TIMEOUT_TEST_BINARY_ID_KIND`**: the kind component of the binary ID, if known.
+
+<!-- TODO: a protocol for writing script logs to a file and telling nextest to attach them to JUnit reports? -->
diff --git a/site/src/docs/configuration/setup-scripts.md → site/src/docs/scripts/setup.md b/site/src/docs/configuration/setup-scripts.md → site/src/docs/scripts/setup.md
@@ -12,77 +12,38 @@ status: experimental
     - **Enable with:** Add `experimental = ["setup-scripts"]` to `.config/nextest.toml`
     - **Tracking issue:** [#978](https://github.com/nextest-rs/nextest/issues/978)
 
-Nextest supports running _setup scripts_ before tests are run. Setup scripts can be scoped to
-particular tests via [filtersets](../filtersets/index.md).
+Nextest runs *setup scripts* before tests are run.
 
-Setup scripts are configured in two parts: _defining scripts_, and _setting up rules_ for when they should be executed.
+## Defining setup scripts
 
-## Defining scripts
+Setup scripts are defined using the top-level `script.setup` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
 
-Setup scripts are defined using the top-level `script` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
-
-```toml title="Setup script definition in <code>.config/nextest.toml</code>"
-[script.my-script]
+```toml title="Script definition in <code>.config/nextest.toml</code>"
+[script.setup.my-script]
 command = 'my-script.sh'
 ```
 
-Commands can either be specified using Unix shell rules, or as a list of arguments. In the following example, `script1` and `script2` are equivalent.
+See [_Defining scripts_](index.md#defining-scripts) for options that are common to all scripts.
 
-```toml
-[script.script1]
-command = 'script.sh -c "Hello, world!"'
-
-[script.script2]
-command = ['script.sh', '-c', 'Hello, world!']
-```
+Setup scripts support the following additional configuration options:
 
-Setup scripts can have the following configuration options attached to them:
-
-- **`slow-timeout`**: Mark a setup script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, setup scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
-- **`leak-timeout`**: Mark setup scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
 - **`capture-stdout`**: `true` if the script's standard output should be captured, `false` if not. By default, this is `false`.
 - **`capture-stderr`**: `true` if the script's standard error should be captured, `false` if not. By default, this is `false`.
 
 ### Example
 
 ```toml title="Advanced setup script definition"
-[script.db-generate]
+[script.setup.db-generate]
 command = 'cargo run -p db-generate'
-slow-timeout = { period = "60s", terminate-after = 2 }
-leak-timeout = "1s"
 capture-stdout = true
 capture-stderr = false
 ```
 
-## Setting up rules
-
-In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can set up a script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
-
-```toml title="Basic rules"
-[[profile.default.scripts]]
-filter = 'rdeps(db-tests)'
-setup = 'db-generate'
-```
-
-(This example uses the `rdeps` [filterset](../filtersets/index.md) predicate.)
-
-Setup scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
+## Specifying setup script rules
 
-```toml title="Platform-specific rules"
-[[profile.default.scripts]]
-platform = { host = "cfg(unix)" }
-setup = 'script1'
-```
-
-A set of scripts can also be specified. All scripts in the set will be executed.
-
-```toml title="Multiple setup scripts"
-[[profile.default.scripts]]
-filter = 'test(/^script_tests::/)'
-setup = ['script1', 'script2']
-```
+See [_Specifying rules_](index.md#specifying-rules).
 
-## Script execution
+## Setup script execution
 
 A given setup script _S_ is only executed if the current profile has at least one rule where the `filter` and `platform` predicates match the current execution environment, and the setup script _S_ is listed in `setup`.