Use source maps

[jrodriguesimg]

Hi,

I don't really understand how this package works so I apologise in advance if this isn't the right place to make this question.

On my project, the code that reaches the browser is processed by babel. The final code has source maps that google chrome uses to show the original source when I open the dev tools. However the puppeteer coverage information of a file shows the processed source code, not the original. Is there a way to convert the text and ranges fields of the coverage object to the original source and line numbers?

Thank you for your antention

[vkoves]

Hm, I know very little about this. @bcoe - does Istanbul natively handle source maps? Is there a way we could do this?

[beaugunderson]

I'm also looking for this; my next step is to investigate babel-plugin-istanbul.

edit: hmm, not what we're looking for, I think.

[vegarringdal]

It would be really nice if we could get code coverage from source maps used.
Maybe a option of files/paths to only use too.
This way we could get coverage of the lib you are using only/and want coverage report of.

[smokku]

@beaugunderson babel-plugin-istanbul is what you are looking for, although extracting coverage information from browser is a bit convoluted.

Istanbul stores coverage data in __coverage__ global, so you need to extract it from browser window and convert to nyc format. It is actually quite easy with Puppeteer: https://github.com/smokku/wrte/blob/62059a4/test/runner.js#L36

BTW: I tried replacing babel-plugin-istanbul with puppeteer-to-istanbul, but lack of source maps support brought me to this issue and caused reverting back to babel-plugin-istanbul.

[aadityataparia]

If files have sourcemaps, they will be linked when you run nyc coverage

[stevenvachon]

This is sorely needed.

@aadityataparia there is no nyc coverage listed in the CLI help. I've been using nyc report and it is not including source maps where I use puppeteer.

It might be related to puppeteer/puppeteer#3570

[bcoe]

@stevenvachon the newest version of v8-to-istanbul which is the library that remaps coverage from V8 format to Istanbul format now supports source-maps, perhaps try experimenting with this?

[felixfbecker]

Without this, the tool is not really useable unfortunately if you're using any bundler like webpack :/

[stevenvachon]

You can avoid this problem by using c8 in place of nyc and excluding this library.

[SergeyPirogov]

@stevenvachon could you please tell how to use c8 instead of nyc?

[stevenvachon]

@SergeyPirogov I will have to retract my above statement due to bcoe/c8#162

[SergeyPirogov]

@bcoe could you please help, I see that puppeteer to Istanbul use v8 to istanbul under the hood. How to map coverage to the original source?

[lukeapage]

So.. I've been trying to get this working.

I re-wrote this package to use the latest v8-to-istanbul (which "supports" sourcemaps)

Sorry but this is typescript and in POC stage..

const fs = require("fs");
const mkdirp = require("mkdirp");
const path = require("path");
const v8toIstanbul = require("v8-to-istanbul");
const convertSourceMap = require("convert-source-map");

class PuppeteerToIstanbul {
  coverageInfo: Array<any>;

  constructor(coverageInfo: any) {
    this.coverageInfo = coverageInfo;
  }

  convertCoverage(coverageItem: any) {
    return [
      {
        ranges: coverageItem.ranges.map(this.convertRange),
        isBlockCoverage: true
      }
    ];
  }

  // Takes in a Puppeteer range object with start and end properties and
  // converts it to a V8 range with startOffset, endOffset, and count properties
  convertRange(range: { start: number; end: number }) {
    return {
      startOffset: range.start,
      endOffset: range.end,
      count: 1
    };
  }

  async writeIstanbulFormat({
    sourceRoot,
    servedBasePath
  }: {
    sourceRoot?: string;
    servedBasePath?: string;
  }) {
    mkdirp.sync("./.nyc_output");

    for (let index = 0; index < this.coverageInfo.length; index++) {
      const coverageInfo = this.coverageInfo[index];

      const sourceMap =
        convertSourceMap.fromSource(coverageInfo.text) ||
        convertSourceMap.fromMapFileSource(coverageInfo.text, servedBasePath);

      const script = v8toIstanbul(
        path.join(sourceRoot || "", `original_downloaded_file_${index}`),
        0,
        {
          source: coverageInfo.text,
          sourceMap
        }
      );
      await script.load();
      script.applyCoverage(this.convertCoverage(coverageInfo));

      const istanbulCoverage = script.toIstanbul();
      Object.keys(istanbulCoverage).forEach(file => {
        if (
          file.indexOf("original_downloaded_file_") >= 0 ||
          file.indexOf("node_modules") >= 0
        ) {
          delete istanbulCoverage[file];
        }
      });

      if (Object.keys(istanbulCoverage).length > 0) {
        fs.writeFileSync(
          `./.nyc_output/out_${index}.json`,
          JSON.stringify(istanbulCoverage)
        );
      }
    }
  }
}

export default function(
  coverageInfo: any,
  options?: { sourceRoot?: string; servedBasePath?: string }
) {
  const pti = new PuppeteerToIstanbul(coverageInfo);
  return pti.writeIstanbulFormat(options || {});
}

However I then discovered that v8-to-istanbul doesn't support sourcemaps with multiple files - meaning that you can't use it with bundles.

So I started a POC to add that support to v8-to-istanbul - I've made some progress note its in an extremely hacky stage: lukeapage/v8-to-istanbul@c169da3

I gave up trying to make it work with a minifier - the mappings were too extreme. Without minification it seems to be creating the right ranges in the right files, but its not doing very well at converting those ranges into branches, lines and statements - I seem to always end up with 100% coverage.

I'm pretty close to giving up on v8 coverage and just instrumenting with istanbul..

[lukeapage]

So it turns out that the report given by pupeteer and playwright is just missing too much information to be used with with pupeteer-to-istanbul.

I copied the coverage class from playwright, fixed the imports back to the right place and changed it to return the raw v8 information. piping that into v8-to-istanbul directly gives good coverage.

[bcoe]

So it turns out that the report given by pupeteer and playwright is just missing too much information to be used with with pupeteer-to-istanbul.

@lukeapage interesting, what bumps into issues?

also thanks for digging into this, unfortunately I write almost no frontend code these days and have been a poor steward of this library.

[lukeapage]

Because it merges ranges it squashed branches and functions. Because callCount is false, you don’t get 0 count ranges so defaulting the line count to 1 means 100% line coverage

Looks like playwright at least will soon expose the raw v8 info.

[Quramy]

I have same issue and the above @lukeapage @bcoe 's comments help me much.

Because it merges ranges it squashed branches and functions. Because callCount is false, you don’t get 0 count ranges so defaulting the line count to 1 means 100% line coverage

Looks like playwright at least will soon expose the raw v8 info.

Puppeteer edits raw V8 coverage result via https://github.com/puppeteer/puppeteer/blob/main/src/common/Coverage.ts#L382 .
And I found v8-to-istanbul sometimes can't map the range with source-map because the joined range can straddles original V8 range associated with some original source and another range not associated.

I think Puppeteer should expose raw V8 script coverages as Playwrite does. puppeteer/puppeteer#6454

[gaitat]

Is there a solution to this issue or any alternatives?

[Izhaki]

@smokku suggestion worked for me.

Background

• Create React App (babel config patched with craco)
• Source files are typescript

Steps

• Add babel-plugin-istanbul to babel config
• Run the app
• Run tests with Puppeteer

Then based on the link @smokku provided:

AfterAll(async function () {
  const outputDir = "../.nyc_output";
  // eslint-disable-next-line no-underscore-dangle
  const coverage = await page.evaluate(() => window.__coverage__);

  await fs.emptyDir(outputDir);
  await Promise.all(
    Object.values(coverage).map(cov => {
      if (
        cov &&
        typeof cov === "object" &&
        typeof cov.path === "string" &&
        typeof cov.hash === "string"
      ) {
        return fs.writeJson(`${outputDir}/${cov.hash}.json`, {
          [cov.path]: cov,
        });
      }
      return Promise.resolve();
    })
  );

  // Make sure the browser is closed
  if (browser != null) {
    browser.close();
  }
});

That generates the .nyc_output folder (under project root).

Then in the project root:

npx nyc report --reporter=html --reporter=text

Result

-------------------------------------------------------|---------|----------|---------|---------|------------------------------------------------------------------------------------------------
File                                                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                              
-------------------------------------------------------|---------|----------|---------|---------|------------------------------------------------------------------------------------------------
All files                                              |   16.29 |     8.52 |    6.02 |   16.73 |                                                                                                
 src                                                   |     100 |      100 |     100 |     100 |                                                                                                
  index.tsx                                            |     100 |      100 |     100 |     100 |                                                                                                
 src/app                                               |   80.49 |    59.09 |   76.92 |   80.56 |                                                                                                
  App.tsx                                              |   66.67 |       75 |      75 |   63.16 | 10-11,15-19                                                                                    
  Routes.tsx                                           |      95 |       50 |      80 |     100 | 49-51,57-62        
...

Versions Used

"babel-plugin-istanbul": "^6.1.1",
"nyc": "^15.1.0",

[rschoenbichler]

Is there any solution that works with webpack bundler as well?

[umarkhanTR]

Is there any solution for this?