Screenshot failure not failing on CI

[joekrump]

I have a screenshot failure that I get when I run my tests in my docker container on my local machine and it's a legitimate failure; Something in the UI changed. However, when the test is run on CI, it does not report any failures even though I can see that it outputs a and "actual" and "diff" file for the .toMatchImageSnapshot() expectation that fails. So it is recognizing that something is different but vitest isn't reporting a failure for some reason.

Is there any sort of special treatment of toMatchImageSnapshot() failures based on environment variables or something like that? That's the only thing I can think of. Maybe something based on whether CI is set or not?

[unional]

Did you await the call?
It is an async operation.

[joekrump]

Did you await the call? It is an async operation.

Yes. It is awaited and fails 💯 of the time when running in my container locally and even when running npx vitest on my local machine directly. It just does not result in a failure on CI. It's super strange and the only thing I've been able to think that could be different is some environment variable.

On CI it is simply as thought the fact that there is a diff isn't treated as a failure for some reason since the file in the __diff__ dir is there and shows the diff I'm expecting. It just doesn't fail the expectation.

[unional]

That's weird. All it does is just throw an error. Do you have a link to the CI job that passes?

This may be a question the Vitest team can help.

[joekrump]

That's weird. All it does is just throw an error. Do you have a link to the CI job that passes?

This may be a question the Vitest team can help.

So here's the bit from the test test:

When the last line of the test block (it block) is this, and the screenshots don't match, the test still passed:

await expect(page.getByTestId("notes-container").element()).toMatchImageSnapshot();

When I update the test so that there is a failure on the line after the screenshot, it fails:

await expect(page.getByTestId("notes-container").element()).toMatchImageSnapshot();
expect(false).toBe(true); // <- intentional failure

When I add the intentionally failing expectation after the screenshot assertion then the screenshot assertion failure shows up for me on CI:

[unional]

Which version of Vite and Vitest are you using?

I notice the sourcemap is also not correct. I suspect there are some versioning issue. See if you have multiple versions of Vite and Vitest installed also.

[joekrump]

Which version of Vite and Vitest are you using?

I notice the sourcemap is also not correct. I suspect there are some versioning issue. See if you have multiple versions of Vite and Vitest installed also.

Vite is v5.4.7 and vitest is v3.0.5

[joekrump]

I notice the sourcemap is also not correct.

Ya that has been a real pain and I haven't nailed down exactly where the problem is as sourcemaps work correctly for our app in prod. Something is configured slightly differently for vitest and haven't nailed it down yet.

[unional]

@joekrump are you still facing this issue?

[joekrump]

@joekrump are you still facing this issue?

Yes. I have not been able to put more time into investigating the root cause so as of right now I've been running all the tests locally to keep things up to date and catch errors. It's not practical for my team but it's been working okay as a stop-gap solution for now