[e2e] Update Console error fixture to be more informative

[unlikelyzero]

Summary

When the Console error detection fails in the fixtures.js, it's not obvious to users why it failed. We need to fix that with some sort of logging that's apparent in CI.

[unlikelyzero]

Testing Notes

Create a console error in the e2e tests
Verify that the errors are clear in the failure report

Create a console warning in the e2e tests
Verify that the warnings do not appear on the failure report

[jvigliotta]

Unable To Test Locally: Testathon: 7/8/22

Many tests are failing, this is after running: npm run test:e2e

[unlikelyzero]

Unable To Test Locally: Testathon: 7/8/22

Many tests are failing, this is after running: npm run test:e2e

@jvigliotta try running npm run test:e2e:ci because this references a narrower but stable set of tests

[akhenry]

• Added console.error and console.warn in e2e tests. Console messages appeared in the console.
• Threw an error from an e2e test, and verified that it appears in the console, and test failed.

I believe this is verified?

[khalidadil]

@akhenry Do you see the same thing for console.error?

[akhenry]

@khalidadil Typo, I meant console.error.

[khalidadil]

Verified Fixed in Testathon on 07/11/22

[akhenry]

I added a console.error to an otherwise passing test and ran the test suite with npm run test:e2e:ci. Although the error was logged to the console, the test still passed. Is this a failure then?

[khalidadil]

@akhenry I just talked to @unlikelyzero, and you have to do it in the browser scope like this:

        await Promise.all([
            page.evaluate(() => console.error('This should result in a failure')),
            page.waitForEvent('console') // always wait for the event to happen while triggering it!
        ]);

Is that how you did it?

Doing it that way did cause it to fail for me: