Coverage Overwritten after each testsuite

[Bmahecha123]

Hello!

I have a StencilJS project (version 0.14.0) and they have started using Puppeteer, they have provided a small library to implement Puppeteer tests. I have implemented your dependency the way you have documented it. However if I do the same setup in each test file the previous coverage appears to get overwritten. This is reflected when I generate the nyc report. Here is my general setup for each test file.

`import { newE2EPage, E2EPage } from '@stencil/core/testing';
import * as pti from 'puppeteer-to-istanbul';

describe('my-component', () => {
let page: E2EPage;

beforeEach(async () => {
page = await newE2EPage();
await page.coverage.startJSCoverage();
});

afterEach(async () => {
const jsCoverage = await page.coverage.stopJSCoverage();

pti.write(jsCoverage);

});`

Please let me know if there is any more information that you need. Thank you for your time, speak to you soon hopefully!

[ps011]

Maybe you can use beforeAll and afterAll

[joe223]

Every Page instance would generate a new puppeteerFormat. How could we merge multiple puppeteerFormat data?

Ref:

puppeteer-to-istanbul/index.js

Line 4 in c57bd74

write: (puppeteerFormat) => {

Update
We should set jsonPart as a top scoped variable in puppeteer-to-istanbul, so every page instance test coverage record could be stored.

puppeteer-to-istanbul/lib/puppeteer-to-istanbul.js

Lines 35 to 45 in de9109c

	let jsonPart = {}
	jsonPart[keys[0]] = istanbulCoverage[keys[0]]
	jsonPart[keys[0]].originalUrl = jsFile.originalUrl
	let jsonStr = JSON.stringify(jsonPart)
	.replace(/^{/, '')
	.replace(/}$/, '')
	const isLastIteration = index === (this.puppeteerToV8Info.length - 1)
	fs.writeSync(fd, jsonStr + (isLastIteration ? '' : ','))
	})

Besides iterator should be same as jsonPart

puppeteer-to-istanbul/lib/output-files.js

Lines 62 to 66 in de9109c

	this.iterator++
	str = `${parsedPath}-${this.iterator}.js`
	return str
	} else {
	str = `${parsedPath}.js`

And, should not save file as ${parsedPath}-${this.iterator}.js if it is inline.

Update
This is the fix PR #42

[sebnitu]

Was this ever resolved or did anyone find a workaround? I'm having the same issue where I can't find a way to keep all my results from multiple test suites. Do we just have to write all E2E tests in a single suite to get coverage with puppeteer-to-istanbul?

[joe223]

@sebnitu Hi, what's the version you had used? And, would you mind give us a repo for producing this issue?

BTW, make sure that you didn't forget @next tag with installation.

npm i puppeteer-to-istanbul@next

[sebnitu]

@joe223 Thanks for the reply!

You can see this issue in my branch here: https://github.com/sebnitu/scroll-stash/tree/add-coverage-to-puppeteer

This script should reproduce the issue: npm run test:integration

When I install using @next I get v1.3.0, is that correct? Regardless, the issue is the same using v1.4.0.

I currently have 4 integration test suites and depending on which one finishes last, I get that suites coverage results. This is consistent when I run each file individually and the result doesn't change when they're ran together.

Current implementation

Each suite has the following setup:

beforeAll(async () => {
  await page.coverage.startJSCoverage();
  // other stuff..
});

afterAll(async () => {
  const jsCoverage = await page.coverage.stopJSCoverage();
  pti.write(jsCoverage);
});

[joe223]

I guess you should create an independent Page instance to collect coverage data. @sebnitu

eg:

describe('Wheel Action', () => {
    let page = null

    before(async function () {
        page = await browser.newPage()
        await page.coverage.startJSCoverage()
        await page.goto(global.entryPath)
    })

   after(async function () {
        const [jsCoverage] = await Promise.all([
            page.coverage.stopJSCoverage()
        ])

        pti.write([...jsCoverage])
    })
    // balabala

[babaosoftware]

@joe223 @sebnitu The "problem" is in https://github.com/istanbuljs/puppeteer-to-istanbul/blob/master/lib/puppeteer-to-istanbul.js, line 29: fs.writeFileSync(outFilePath, '')
where the out.json file is truncated every time pti.write is called. So depending where you call it, or how many test/test suites you have, you will get only the results for the last one that called pti.write.

[babaosoftware]

@sebnitu To merge coverage information from different test suites, you can do this:

• In your afterAll, when calling pti.write, pass a specific name for the json file like this: await pti.write([...jsCoverage, ...cssCoverage], {outputName: 'my-out-file' }); This will create a file my-out-file.json in .nyc-output directory, just for this test suite.
• After all test suites have run, execute nyc merge .nyc_output ./.nyc_output/coverage.json to merge all the files in one (.nyc_output is the directory where coverage results are written)
• Run nyc report to generate the coverage report.

[babaosoftware]

My suggestion with passing outputName was probably based on a forked version I was working on. Basically I created my own option for PuppeteerToIstanbul, to override the out.json file name, much like it's done for the storagePath, "./.ny_output". Maybe I'll open a pull request for it.

[pea-nut-z]

Every time you open a new page, the coverage gets overwritten. If you keep all the testings on the same page then this wouldn't be a problem. For example, open new page and start coverage in your first describe block's beforeAll function. Then, close page and end coverage in your last describe block's afterAll function. If you need to reset your page in each describe, you could do a refresh page instead of opening a new page that erases previous coverage.
That's a simple solution for people who don't care to start new page in every test block.
I did that for my project below.
https://github.com/pea-nut-z/sneakers-image-classification