Skip to content

Latest commit

 

History

History
195 lines (140 loc) · 13.3 KB

README.md

File metadata and controls

195 lines (140 loc) · 13.3 KB

Playwright code coverage with Istanbul via Webpack Babel plugin

Description

The demo at /demos/code-coverage-with-istanbul-via-webpack-babel-plugin shows how to get code coverage with Playwright by using Istanbul to instrument the code.

We can use the babel-plugin-istanbul Webpack plugin to instrument the code with Istanbul. The instrumented code will automatically generate coverage data and save it on window.__coverage__.

Then this demo saves the code coverage data into files so that after we can generate code coverage reports using the nyc CLI tool.

Note

This solution was implemented based on the work done by @mxschmitt. See his work at https://github.com/mxschmitt/playwright-test-coverage. As of writing this, there's an open GitHub issue about improving the built-in code coverage support for Playwright. Also check out this comment on that issue.

Thank you @mxschmitt !

How to build, run the app and run tests

Important

Required dependencies:

  • Node. Tested working with v20.10.0. If you need to have different versions of node installed it's recommended that you use Node Version Manager to install and swap between node versions.
  • npm@latest: package manager used on the demos. Tested working on 10.2.5.
  • VS Code is recommended as a code editor but you can use whatever you prefer.
  1. Clone the repo.
  2. Using your favorite shell go to /demos/code-coverage-with-istanbul-via-webpack-babel-plugin.
  3. Install the required npm packages with: 
    npm install
  4. Install the playwright browsers with: 
    npx playwright install
  5. Run the tests with: 
    npm test
    This will start the app and run the playwright tests against it.
  6. If you just want to run the app execute the command: 
    npm start
    Once the command finishes the app should open in your default browser at http://127.0.0.1:4200/.

How to view the test results and code coverage

After running the tests with npm test you can view the test results with:

npm run test:show-report

You can generate the code coverage reports with:

npm run coverage:report

And you can view the code coverage html report with:

npm run coverage:show-report
code-coverage-with-istanbul-via-webpack-babel-plugin.mp4

Tests and code coverage

To get code coverage with Istanbul there are two main building blocks:

  • use the babel-plugin-istanbul Webpack plugin to collect code coverage during the tests' execution: npm i -D babel-plugin-istanbul.
  • use the nyc CLI tool to generate the code coverage reports from the code coverage data: npm i -D nyc

Playwright configuration

The majority of the content of the playwright.config.ts file is what you get by default after adding Playwright to your project with npm init playwright@latest.

The main changes are:

  1. Declared a few variables at the start that are reused throughout the playwright configuration.
  2. Updated the reporter array. Added the built-in list reporter to the default html reporter.
  3. Defined a baseURL so that we can use relative URLs when doing page navigations on the tests.
  4. Configured the webServer block to run the Angular app locally so that the tests can be executed against it. If you're not testing an Angular app that's fine, you just need to adjust the webServer.command so that it launches your app and set the webServer.url to the url your app will be running at. For more information see the webServer docs.

Note

The _isRunningOnCI variable used on the playwright.config.ts changes the value of some options when running tests on CI. To set the _isRunningOnCI variable to true you must set the environment variable CI to true before running the tests. For more information regarding using Playwright on a CI environment see Playwright docs on Continuous Integration.

Furthermore, we have created:

Note

You don't have to create the playwright.shared-vars.js, the playwright.cli-options.ts or the playwright.env-vars.ts file. You can have all of this on the playwright.config.ts. Code structure is up to you.

Note

Depending on your playwright.config.ts, make sure you update your .gitignore to exclude any directory used by test results, report results, etc. Scroll to the end of this demo's .gitignore to see an example.

Collect code coverage data

To instrument the code whilst the tests are running we used the babel-plugin-istanbul Webpack plugin. To use this plugin we have to extend Angular's Webpack configuration and this demo uses the @angular-builders/custom-webpack npm package to do that.

By using the @angular-builders/custom-webpack package we can then update the angular.json to indicate that we have an extra Webpack configuration file we want to be executed. See the customWebpackConfig option in the angular.json file.

We set the customWebpackConfig to the coverage.webpack.js file which adds the babel-plugin-istanbul to the Webpack's blunding process.

Note

This code demo uses an Angular 17 app but the concept is still valid for any other frontend framework. What you have to do is extend your blunding process to enable capturing code coverage data and different frontend frameworks will have different ways to allow you to do this.

Furthermore, for those using Angular, this is an Angular 17 app but it's not using the new Angular esbuilder, it's still using Webpack. The reason is that I don't know how to extend the bundling process of an Angular 17 app using esbuild, perhaps I'll give a try again when this GitHub issue just-jeb/angular-builders Support Customization of new application builder #1537 and PR just-jeb/angular-builders feat(custom-esbuild): allow providing ESBuild plugins #1536 are complete.

Now, when the app is running in development mode we will have Istanbul instrumenting the code and saving the coverage data on a window variable, the window.__coverage__.

With the code instrumentation in place, the last part we need to do is to save the collected code coverage data so that later we can generate the code coverage reports. This is done by the istanbul-code-coverage.ts file. The istanbul-code-coverage.ts is invoked from a Playwright automatic fixture. Playwright fixtures allow us to easily reuse logic across tests.

The automatic fixture is implemented at app-fixtures.ts.

With this automatic fixture in place, the code coverage data is saved to files for any test without having to do anything extra as long as the test import comes from this fixture. See the example.spec.ts and note the import statment at the top:

import { test, expect } from "tests/_shared/app-fixtures";

To use the automatic fixture all your tests should import the test from the fixture instead of doing the usual:

import { test, expect } from "@playwright/test";

To learn more about why this is done study how fixtures work on Playwright.

With all of the above in place, after you execute npm test you will see the collected code coverage data saved into the tests/test-results/code-coverage/istanbul-instrumentation folder.

Generate code coverage reports

After running npm test to run the tests and generate the code coverage data, you can run the npm run coverage:report command which will use nyc CLI tool to create the code coverage reports.

The nyc configuration file nyc.config.js will create:

  • an html report with the istanbul html-spa style. This is a human readable code coverage report.
  • an lcov report: which is useful to upload to some tools like SonarQube, etc.
  • a cobertura report: which is useful to upload to some tools like Azure DevOps, CodeCov, etc.
  • a text-summary report: which is useful summary output that you get on the console after running the command to generate code coverage.

The produced folder structure after running command to generate code coverage looks like this:

tests/
└── test-results/
    └── code-coverage/
        ├── istanbul-instrumentation/
        │   └── ... (coverage data from running the tests)
        └── reports/
            ├── index.html (entry point to the html report)
            ├── cobertura-coverage.xml
            ├── lcov.info
            └── ... (other files to support the html report)

Caution

When I added the text-summary reporter to the nyc.config.js I added it as the first element of the reporter array. This however had unintended consequences because it somehow affected the output produced by the html-spa reporter making it invalid. Moving the text-summary to the last element of the reporter array seems to fix this quirk.

Learn more

To learn more about Istanbul and configuring the nyc reporter see:

JS, CSS and HTML code coverage

Unlike the demo that uses V8 to get code coverage, this demo is only capable of collecting JS code coverage, it won't provide CSS or HTML coverage.