diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 827b66e2..6347d834 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -50,6 +50,7 @@ - [Resources](./compiler/resources.md) - [Review Policy](./compiler/reviews.md) - [Proposals, Approval and Stabilization](./compiler/proposals-and-stabilization.md) + - [Adding ecosystem/integration test jobs/components to `rust-lang/rust` CI](./compiler/proposals-and-stabilization/custom-test-jobs.md) - [Third-party and Out-of-tree Crates Policy](./compiler/third-party-out-of-tree.md) - [Triage and Prioritization](./compiler/prioritization.md) - [Operations](./compiler/operations.md) diff --git a/src/compiler/proposals-and-stabilization.md b/src/compiler/proposals-and-stabilization.md index 946e5ece..01b3a9ab 100644 --- a/src/compiler/proposals-and-stabilization.md +++ b/src/compiler/proposals-and-stabilization.md @@ -329,9 +329,15 @@ circumstance. - **Approve using:** r+ (compiler leads) - Open a PR with the change affecting licensing and assign it to the team leads for review +### Adding ecosystem/integration test jobs/components to rust-lang/rust CI + +See [*Adding ecosystem/integration test jobs/components to rust-lang/rust CI*](./proposals-and-stabilization/custom-test-jobs.md). + + [stabilization_guide]: https://rustc-dev-guide.rust-lang.org/stabilization_guide.html [tier_policy]: https://doc.rust-lang.org/rustc/target-tier-policy.html [mcps]: https://github.com/rust-lang/compiler-team/issues?q=label%3Amajor-change [rfcs]: https://github.com/rust-lang/rfcs [rust]: https://github.com/rust-lang/rust [compiler_lint_eg]: https://doc.rust-lang.org/rustc/lints/listing/deny-by-default.html#explicit-builtin-cfgs-in-flags +[ecosystem_testing]: https://rustc-dev-guide.rust-lang.org/tests/ecosystem.html diff --git a/src/compiler/proposals-and-stabilization/custom-test-jobs.md b/src/compiler/proposals-and-stabilization/custom-test-jobs.md new file mode 100644 index 00000000..37c5db13 --- /dev/null +++ b/src/compiler/proposals-and-stabilization/custom-test-jobs.md @@ -0,0 +1,248 @@ +# Adding ecosystem/integration test jobs/components to rust-lang/rust CI + +## Scope + +This policy is applicable to proposals for adding new ecosystem and integration test jobs/components +that involve building and testing additional artifacts which may cause [rust-lang/rust] PR CI or +Full CI (the "CI") to fail, or produce a failure message, that may impact other [rust-lang/rust] +contributors who also use the [rust-lang/rust] CI. + +For example (non-exhaustive): + +- Ecosystem test jobs: Rust for Linux or Fuchsia. +- Integration test components: GCC codegen backend or Cranelift codegen backend. + +## Background + +[rust-lang/rust] runs a small set of build/test jobs on PR CI (faster, less expensive/exhaustive), +and runs a much larger set of build/test jobs on Full CI. PR CI usually takes around an hour, while +Full CI usually takes around 3 hours. Having ecosystem and integration test jobs/components that: + +- may spuriously fail; or +- may genuinely fail but it's not clear who should be consulted about the failure or who is + responsible for fixing the failure; or +- otherwise do not have well documented test job maintainers and failure protocol; + +can introduce a lot of friction and frustration for other contributors who now will have to deal +with the failure or understand why the test job failed in the first place, which may now be blocking +their PR. Everyone utilizing the [rust-lang/rust] CI is expected to use it responsibly. + +To help with this, please follow the process described below if you would like to propose adding an +ecosystem/integration test job/component to the [rust-lang/rust] CI. + +## Process for adding an ecosystem/integration test job/component to rust-lang/rust CI + +![Flowchart of the process for adding an ecosystem/integration test job/component to rust-lang/rust +CI](./custom-test-jobs/test-job-flow.svg) + +1. Ask the Infrastructure Team on zulip + ([#t-infra](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra)) to check if there + would be capacity for the proposed test job/component. +1. Propose using a [Major Change Proposal + (MCP)](../proposals-and-stabilization.md#how-do-i-submit-an-mcp). + - Including the filled out *Ecosystem and Integration Test Job/Component Policy* (see below). +1. Once the MCP is seconded and accepted, create a new issue on [rust-lang/rust] about the proposal, + linking to the MCP, and then nominate the proposal for library team review via `@rustbot label + +I-libs-nominated`. + - Link to the MCP. +1. Once the library team reviews the proposal and has no blocking concerns, submit the + implementation PR to [rust-lang/rust]. + - The PR should include an accompanying Ecosystem/Integration Test Job/Component support page in + the in-tree [rustc-dev-guide], ensuring that the *Failure Protocol* is well-documented (see + below). Link to the MCP and issue. +1. The Infrastructure Team will review the implementation PR. Once approved, the + ecosystem/integration test job/component will run in [rust-lang/rust] PR CI or Full Merge CI. + +## Ecosystem and Integration Test Job/Component Policy + +Please copy and fill out this template (below the separator) as part of the MCP. The policy +questions/explanation themselves will be quoted blocks. Information that the MCP author is expected +to provide are indicated by italicized sentences whose content should be replaced. + +NOTE: It is not the intention of this policy to make it frustrating annoying to add an ecosystem +test job/component. We simply wish to gather the necessary background information upfront +(especially working together to figure out a viable *Failure Protocol*) to minimize potential +frustrations from other [rust-lang/rust] contributors if and when the the test job/component do +fail, potentially blocking PR / Full Merge CI in completely unrelated PRs. + +--- + +```markdown +## Ecosystem and Integration Test Job/Component Policy + +The ecosystem/integration test job/component ("test job/component") proposed for the +[rust-lang/rust] CI must: + +- Be approved by the compiler team through a proposed MCP, where the MCP is seconded by a compiler + team member, and the MCP is accepted with no blocking concerns. +- Have no blocking concerns from the library team. +- Have the implementation PR be reviewed and approved by the infrastructure team. +- Be properly documented on [rustc-dev-guide] (preferably as part of the implementation PR). + +Please complete the sections below so [rust-lang/rust] teams can have sufficient context about the +proposed test job/component. + +### Test job/component rationale + +> What does this test job/component do? +> +> - If an ecosystem test job/component is being proposed, can you briefly describe the intended +> ecosystem users? + +*Please provide responses here, replacing this sentence.* + +> What [rust-lang/rust] changes can potentially break the test job/component? +> +> E.g. changes to rustc, standard library, bootstrap or tools (like clippy/rustfmt/cargo). + +*Please provide responses here, replacing this sentence.* + +> Why does this test job/component need to be part of the [rust-lang/rust] PR and/or Full Merge CI? + +*Please provide responses here, replacing this sentence.* + +> If the test job/component will block on failure, why does it need to block? + +*Please provide responses here, replacing this sentence.* + +> If the test job/component will not block on failure initially but is intended to eventually become +> blocking: +> +> - Why will it become blocking? +> - When will it become blocking? + +*Please provide responses here, replacing this sentence.* + +### Test job/component maintainers + +> The proposed test job/component for [rust-lang/rust] CI must have at least one dedicated test +> job/component maintainer. The test job/component maintainers understand that they will be pinged +> or otherwise contacted about the custom test job/component, particularly for (but not limited to) +> its failures. +> +> **Please list who will be maintaining this ecosystem/integration test job/component here. Please +> format the github handles in the style:** +> +> ``` +> [@github_handle_1](https://github.com/github_handle_2) +> [@github_handle_2](https://github.com/github_handle_2) +> ``` +> +> NOTE: For future readers, you can paste the usernames without formatting them as links via +> **ctrl-shift-v**. + +*Please list test job/component maintainers here with the formatting advice above, replacing this +sentence.* + +> **NOTE: If an ecosystem/integration test job/component no longer has an active dedicated +> maintainer (or maintainers), and if [rust-lang/rust] teams find the ecosystem/integration test +> job/component causes significant burden or becomes irrelevant, then the ecosystem/integration test +> job/component may be removed.** + +### CI infrastructure considerations + +> You should ask the Infrastructure Team on the +> [`#t-infra`](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra) zulip channel when +> proposing a new ecosystem/integration test job/component to check if there's capacity for the test +> job/component. +> +> - Does the custom test job/component require substantial CI resources (storage and/or CI time)? In +> particular, will it require large runners? + +*Please provide responses here, replacing this sentence. If there is a zulip topic discussing it +with the Infrastructure Team, please include the zulip topic link here.* + +### Features and implementation details + +> Does the proposed test job/component intend to use any unstable features? +> +> - If so, are the unstable features ready for exposure (e.g. must an unstable feature be completely +> reworked)? +> - For ecosystem test jobs/components, are the unstable features ready for such exposure to the +> ecosystem, and are the feature stakeholders ready for such usage? + +*Please provide responses here, replacing this sentence.* + +> Does the proposed test job/component intend to intentionally depend on any implementation details? +> This may include but is not limited to: unstable/internal compiler/tool flags and behaviors, +> `RUSTC_BOOTSTRAP` usages, standard library implementation details, etc. +> +> - If so, are there plans to shrink or expand such dependencies in the future? + +*Please provide responses here, replacing this sentence.* + +### Failure protocol: what to do if the job/component breaks/fails? + +> **NOTE: If the artifacts of an ecosystem/integration test job/component are not shipped as part of +> a distribution component/toolchain, the test job/component may be temporarily disabled to unblock +> [rust-lang/rust] PR CI or Full Merge CI without receiving prior approval from the test +> job/component maintainers. The test job/component maintainers will be pinged or otherwise notified +> about the test job/component being disabled.** + +> How can the test job/component maintainers be contacted in case of failure? By default, it is +> assumed that the test job/component maintainer can be pinged via their GitHub handles. + +*Please provide responses here, replacing this sentence.* + +> (If applicable) If the addition of an ecosystem/integration test job is being proposed: +> +> - How can the test job be run in CI? If so, is there a try job (`try-job: ...`) invocation? What's +> the job name? +> - Can the test job be run locally? If so, how? + +*If applicable, please provide responses here, replacing this sentence. Otherwise, you can ignore +this question.* + +> (If applicable) If the addition of an ecosystem/integration test component is being proposed: +> +> - Which existing CI jobs will be building and testing this test component? +> - Can they be built and ran as part of a try job? If so, what are the job names and the try job +> (`try-job: ...`) invocations? +> - Can the test component be built and run locally? If so, how? + +*If applicable, please provide responses here, replacing this sentence. Otherwise, you can ignore +this question.* + +> How can the test job/component be disabled in the event of spurious failures that are blocking PR +> and/or Full Merge CI? + +*Please provide responses here, replacing this sentence.* + +> If a PR breaks the test job/component: +> +> - If the breakage seems **spurious** and retrying does not resolve the spurious breakage, the test +> job may be **temporarily disabled** (see below). +> - If the breakage is **intentional**, how will this be resolved? +> - If the breakage is **unintentional**, is the PR author expected to fix the breakage? + +*Please provide responses here, replacing this sentence.* + +### Dependencies, build/test environments and reliability + +> Does the test job/component involve any custom build systems that are not used in the regular +> [rust-lang/rust] CI jobs? + +*Please provide responses here, replacing this sentence.* + +> Does the test job/component depend on external resources (e.g. external servers) that may be +> subject to network connectivity? +> +> - If so, does the infrastructure team need to help maintain a mirror of the required assets? + +*Please provide responses here, replacing this sentence.* + +> Are there any potential sources of spurious failures due to the test job/component? + +*Please provide responses here, replacing this sentence.* + +> Are there any other unusual requirements (build environment, dependencies, etc.)? + +*Please provide responses here, replacing this sentence.* + + +[rust-lang/rust]: https://github.com/rust-lang/rust +[rustc-dev-guide]: https://github.com/rust-lang/rustc-dev-guide +``` + +[rust-lang/rust]: https://github.com/rust-lang/rust +[rustc-dev-guide]: https://github.com/rust-lang/rustc-dev-guide diff --git a/src/compiler/proposals-and-stabilization/custom-test-jobs/test-job-flow.svg b/src/compiler/proposals-and-stabilization/custom-test-jobs/test-job-flow.svg new file mode 100644 index 00000000..9ba45eb8 --- /dev/null +++ b/src/compiler/proposals-and-stabilization/custom-test-jobs/test-job-flow.svg @@ -0,0 +1,3 @@ + + +
Proposing a new ecosysten/integration test job/component for rust-lang/rust CI
Proposing a new ecosysten/integration test job...
Propose using a Major Change Proposal (MCP)
Propose using a Major Change Proposal (MCP)
Compiler team reviews MCP;
Approval via second
Compiler team reviews MCP;...
FCP completes with no outstanding concerns;
MCP is accepted
FCP completes with no outstanding concerns;...
Resolve outstanding concerns
Resolve outstanding concerns
MCP enters Final Comment Period (MCP)
MCP enters Final Comment Period (MCP)
Outstanding concerns
Outstanding concerns
Open a new issue on rust-lang/rust, linking to the MCP, to nominate for library team decision
Open a new issue on rust-lang/rust, linking to...
Outstanding concerns
Outstanding concerns
Outstanding concerns
Outstanding concerns
Library team has no outstanding concerns
Library team has no outstanding concerns
Library team reviews proposal
Library team reviews proposal
Re-nominate for library team if needed
Re-nominate for library team if needed
Resolve outstanding concerns
Resolve outstanding concerns
Submit implementation PR including
rustc-dev-guide Ecosystem/Integration Test Job/Component support page
Submit implementation PR including...
Infrastructure reviewer approval
Infrastructure reviewer approval
Outstanding feedback
Outstanding feedback
Standard Infrastructure team review
Standard Infrastructure team review
Ecosystem/Integration test job/component now runs in PR CI and/or Full Merge CI
Ecosystem/Integration test job/component now r...
Wait for infra reviewer to re-review
Wait for infra reviewer to re-review
Resolve review feedback / requested changes
Resolve review feedback / request...
Ask the Infrastructure team on zulip (#t-infra) if there is capacity for the test job/component
Ask the Infrastructure team on zulip (#t-infra...
For test jobs, typically fine if the job runs within e.g. 80% of the total CI duration on a free runner, but if a large runner is required then further discretion is needed
For test jobs, typically fine if the job runs within...
Including the filled out Ecosystem/Integration Test Job/Component template
Including the filled out Ecosystem/Integration Test J...
Nominate via @rustbot label +I-libs-nominated
Nominate via @rustbot label +I-libs-nominated
Ensure that the Test Job/Component Failure Protocol is well-documented
Ensure that the Test Job/Component Failure Protocol i... \ No newline at end of file