You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a belated "final" version of a planning document that was started back in January/Feburary. Some of the work is already done (✅ sections, done by a mix of Elizabeth, Ling, Tristan, and Martin) but various longer-term items remain.
Overview
We have various layers of testing in Fluent UI, but a lot of the tools are out of date, and there are ways we could make our tests easier to write, faster to run, and generally more effective. These issues reduce development velocity and quality/quantity of tests.
This project's goal is to "clean house," making changes that will facilitate writing great tests to help take our library's quality to the next level as we continue to build out more converged components.
General areas:
Getting up to date: Upgrade Jest, Cypress, and other testing-related deps to the latest versions (and help unblock upgrades of other deps such as React)
Screener: Make our visual tests faster and more reliable, update outdated Storybook usage, and investigate whether a different visual testing framework would serve us better in the long run
Migrate away from Enzyme: Enzyme is effectively dead, was delaying our repo's internal React 17 upgrade, and will entirely block an in-repo React 18 upgrade. We need to work on switching to Testing Library and Cypress, which provide better-supported, less buggy, and more realistic testing.
Improve conformance tests: Conformance tests enable us to test both current and future components, and maintain API/behavior coherency as we build more components. The @fluentui/react-conformance package had ambitious goals, but its current structure is holding us back, so it needs to be rewritten (and then we need to add more tests).
Documentation: We need up-to-date documentation to help people write great tests.
Address "paper cuts": Eliminate or improve small things that are detracting from the developer experience (with respect to testing)
Future ideas: Some additional types of tests we'd like to add at some point in the future
Getting up to date
Our testing-related dependencies had gotten pretty badly out of date, and new versions continue to be released.
✅ Done
Updated Jest from 24 to 26
Updated testing-library to latest as of January 2022
For various reasons (outlined in #21070), our screener runs were becoming less reliable as we added more v9 tests. Also, having v8 and v9 in the same package was slowing down builds (due to needing to build both + run tests for both when either changed).
Made custom screener proxy support multiple projects
Moved v9 screener tests to a separate package from v8 screener tests, and run them via custom proxy
Only run screener for affected projects
Medium/long-term
The storybooks in our vr-tests packages are using an outdated story format, which will need to be fixed before we can migrate to storybook 7 (once it comes out). #21779
We also haven't been impressed with various aspects of screener, so longer-term we'd like to investigate other solutions (possibly chromatic if we can find an interaction testing solution). #22369
Migrate away from Enzyme
Enzyme is no longer in active development (no official React 17 adapter after 1.5 years) and won't work with React 18 without a substantial rewrite of the adapter code/helpers. It's also just kind of difficult to work with and buggy.
A better solution is to use a combination of React Testing Library and Cypress.
Testing Library provides a more user-centric way of testing which often makes it easier to write tests, plus it's less buggy (a bit more closely resembles the browser environment) and makes cleanup easier.
Cypress allows testing in a real browser, for scenarios such as focus which are just not simulated realistically enough by jsdom.
Prior to this work, all the tests for v8 and v0 used Enzyme, and so did the react-conformance and a11y-testing packages shared between versions. v9 did not use Enzyme aside from conformance/a11y.
✅ Done
Goal: do enough to unblock migration to React 17 (#20145). This involves a hybrid approach using the "community" adapter @wojtekmaj/enzyme-adapter-react-17 and porting any tests that don't work to either testing-library or cypress.
Got rid of Enzyme in our testing helpers (react-conformance, a11y-testing)
Enabled Cypress testing for v8 - this provides an "out" for especially tricky tests like FocusTrapZone, or if we upgrade react and/or jest and any other tests stop working even in testing-library
Tried out React 17 with the community adapter and fixed the handful of broken tests (either made small fixes or ported to testing-library or cypress)
At this point the React 17 upgrade should be unblocked from a testing standpoint
We also communicated to the Redmond team that all new v8 tests should be written in testing-library (and strongly encouraged that if you have to fix an enzyme test, go ahead and port it). Ideally this would be enforced by tooling, but this is more challenging since a lint rule can't easily be applied to new code only.
Medium/long-term
Completely get rid of Enzyme by porting all tests (v8 and v0) to either testing-library or cypress. As mentioned above, this will be necessary before we can adopt React 18 in-repo. #21663
An alternative might be finding a way to run only the Enzyme tests against React 17, but that would be challenging to set up.
Conformance tests
Component conformance tests are important because they enable us to test both current and future components, and help us ensure that we have a coherent API surface and behaviors as we continue to build more components.
Short-term
@fluentui/react-conformance was written with the goal of being reusable and extensible, but it's not a very good design. It also has the issue of including tests in the core package that are actually specific to v8/v9/v0 (which can cause dependency graph issues and is generally unhelpful), and it introduced enzyme+node types into the global namespace anywhere it's imported.
Removed Enzyme, so the package no longer causes global type pollution
For the future, it would be good to continue expanding those docs with more information about best practices, "gotchas," etc.
Address "paper cuts"
There are various things creating small slowdowns or inconveniences which detract from the developer experience.
When we encounter one of these things, we should consider "is this thing still providing value? (if not, how hard is it to remove?) or "is there an easy way to make this a little better?"
✅ Short-term
Eliminated example snapshots in v7/8: nobody on the Redmond team could think of a specific instance where an example snapshot caught a real issue, so the snapshots were mostly just an annoyance and a source of confusion for contributors
Made assorted improvements to error messages
Deferred/ideas
Improve reliability of Cypress tests: currently the cypress tests (especially in v0 but sometimes other places) are the most frequent cause of random PR build failures, which is annoying
Fix ID churn issues in v8 snapshots, possibly by calling resetIds() in a global afterEach
Future ideas
Longer-term, what are some ways we can take our tests to the next level?
Add memory tests
Perf test improvements (needs a new owner)
Use the same tool for v8/9/0
Bundle size test improvements (needs a new owner)
Use the same tool for v8/9/0
Ability to have a report link to share out
Partner integration tests
The text was updated successfully, but these errors were encountered:
Overview
We have various layers of testing in Fluent UI, but a lot of the tools are out of date, and there are ways we could make our tests easier to write, faster to run, and generally more effective. These issues reduce development velocity and quality/quantity of tests.
This project's goal is to "clean house," making changes that will facilitate writing great tests to help take our library's quality to the next level as we continue to build out more converged components.
General areas:
Getting up to date: Upgrade Jest, Cypress, and other testing-related deps to the latest versions (and help unblock upgrades of other deps such as React)
Screener: Make our visual tests faster and more reliable, update outdated Storybook usage, and investigate whether a different visual testing framework would serve us better in the long run
Migrate away from Enzyme: Enzyme is effectively dead, was delaying our repo's internal React 17 upgrade, and will entirely block an in-repo React 18 upgrade. We need to work on switching to Testing Library and Cypress, which provide better-supported, less buggy, and more realistic testing.
Improve conformance tests: Conformance tests enable us to test both current and future components, and maintain API/behavior coherency as we build more components. The
@fluentui/react-conformance
package had ambitious goals, but its current structure is holding us back, so it needs to be rewritten (and then we need to add more tests).Documentation: We need up-to-date documentation to help people write great tests.
Address "paper cuts": Eliminate or improve small things that are detracting from the developer experience (with respect to testing)
Future ideas: Some additional types of tests we'd like to add at some point in the future
Getting up to date
Our testing-related dependencies had gotten pretty badly out of date, and new versions continue to be released.
✅ Done
Medium/long-term
@testing-library/user-event
to 14 (bug fixes and API changes) - Upgrade to user-event v14 #22260Screener (vr-tests)
✅ Done
For various reasons (outlined in #21070), our screener runs were becoming less reliable as we added more v9 tests. Also, having v8 and v9 in the same package was slowing down builds (due to needing to build both + run tests for both when either changed).
Medium/long-term
The storybooks in our vr-tests packages are using an outdated story format, which will need to be fixed before we can migrate to storybook 7 (once it comes out). #21779
We also haven't been impressed with various aspects of screener, so longer-term we'd like to investigate other solutions (possibly chromatic if we can find an interaction testing solution). #22369
Migrate away from Enzyme
Enzyme is no longer in active development (no official React 17 adapter after 1.5 years) and won't work with React 18 without a substantial rewrite of the adapter code/helpers. It's also just kind of difficult to work with and buggy.
A better solution is to use a combination of React Testing Library and Cypress.
Prior to this work, all the tests for v8 and v0 used Enzyme, and so did the react-conformance and a11y-testing packages shared between versions. v9 did not use Enzyme aside from conformance/a11y.
✅ Done
Goal: do enough to unblock migration to React 17 (#20145). This involves a hybrid approach using the "community" adapter
@wojtekmaj/enzyme-adapter-react-17
and porting any tests that don't work to either testing-library or cypress.We also communicated to the Redmond team that all new v8 tests should be written in testing-library (and strongly encouraged that if you have to fix an enzyme test, go ahead and port it). Ideally this would be enforced by tooling, but this is more challenging since a lint rule can't easily be applied to new code only.
Medium/long-term
Completely get rid of Enzyme by porting all tests (v8 and v0) to either testing-library or cypress. As mentioned above, this will be necessary before we can adopt React 18 in-repo. #21663
An alternative might be finding a way to run only the Enzyme tests against React 17, but that would be challenging to set up.
Conformance tests
Component conformance tests are important because they enable us to test both current and future components, and help us ensure that we have a coherent API surface and behaviors as we continue to build more components.
Short-term
@fluentui/react-conformance
was written with the goal of being reusable and extensible, but it's not a very good design. It also has the issue of including tests in the core package that are actually specific to v8/v9/v0 (which can cause dependency graph issues and is generally unhelpful), and it introduced enzyme+node types into the global namespace anywhere it's imported.Medium/long-term
Documentation
Our testing documentation was very out of date and didn't cover current recommended tools/patterns.
✅ The main testing page has been revamped and new pages added with specific info about Jest, Cypress, and Screener.
For the future, it would be good to continue expanding those docs with more information about best practices, "gotchas," etc.
Address "paper cuts"
There are various things creating small slowdowns or inconveniences which detract from the developer experience.
When we encounter one of these things, we should consider "is this thing still providing value? (if not, how hard is it to remove?) or "is there an easy way to make this a little better?"
✅ Short-term
Deferred/ideas
resetIds()
in a globalafterEach
Future ideas
Longer-term, what are some ways we can take our tests to the next level?
The text was updated successfully, but these errors were encountered: