Skip to content

Commit

Permalink
comments
Browse files Browse the repository at this point in the history
  • Loading branch information
unlikelyzero committed Nov 9, 2023
1 parent f9c6e82 commit 8967b6a
Showing 1 changed file with 11 additions and 11 deletions.
22 changes: 11 additions & 11 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Documentation located [here](./e2e/README.md)

It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test.

Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in Github PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.
Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in GitHub PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.

### Karma-istanbul

Expand All @@ -63,11 +63,11 @@ Once the file is generated, it can be published to codecov with
### e2e
The e2e line coverage is a bit more complex than the karma implementation. This is the general sequence of events:

1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpacka.coverage.js` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js).
1. Each e2e suite will start webpack with the ```npm run start:coverage``` command with config `webpack.coverage.js` and the `babel-plugin-istanbul` plugin to generate code coverage during e2e test execution using our custom [baseFixture](./baseFixtures.js).
1. During testcase execution, each e2e shard will generate its piece of the larger coverage suite. **This coverage file is not merged**. The raw coverage file is stored in a `.nyc_report` directory.
1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command ```npm run cov:e2e:report```
1. Most of the tests are run in the '@stable' configuration a focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with ```npm run cov:e2e:stable:publish```.
1. The rest of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the ```npm run cov:e2e:full:publish``` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.
1. [nyc](https://github.com/istanbuljs/nyc) converts this directory into a `lcov` file with the following command `npm run cov:e2e:report`
1. Most of the tests are run in the '@stable' configuration and focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`.
1. The rest of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.


### Limitations in our code coverage reporting
Expand All @@ -77,12 +77,12 @@ Our code coverage implementation has some known limitations:
- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973)

## Troubleshooting CI
The following is an evolving guide to trouble CI and PR issues.
The following is an evolving guide to troubleshoot CI and PR issues.

### Github Checks failing
There are a few reasons that your Github PR could be failing beyond simple failed tests.
* Required Checks. We're leveraging required checks in Github so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in Github Checks.
* Not all required checks are run per commit. You may need to manually trigger addition github checks with a `pr:<label>` label added to your PR.
There are a few reasons that your GitHub PR could be failing beyond simple failed tests.
* Required Checks. We're leveraging required checks in GitHub so that we can quickly and precisely control what becomes and informational failure vs a hard requirement. The only way to determine the difference between a required vs information check is check for the `(Required)` emblem next to the step details in GitHub Checks.
* Not all required checks are run per commit. You may need to manually trigger addition GitHub checks with a `pr:<label>` label added to your PR.

### Flaky tests
There are two ways to know if a test on your branch is historically flaky:
Expand All @@ -99,9 +99,9 @@ In certain circumstances, the CircleCI cache can become stale. In order to bust
1. Add Parameter -> Parameter Type = boolean , Name = BUST_CACHE ,Value = true
1. Click 'Trigger Pipeline'

#### Run tests in the same container
#### Run tests in the same container as CI

In extreme cases, tests can fail due to the constraints of running within a container. To execute tests in exactly the same way as run in CircleCI. To date, we've not had github actions fail differently than local execution.
In extreme cases, tests can fail due to the constraints of running within a container. To execute tests in exactly the same way as run in CircleCI.

```sh
// Replace {X.X.X} with the current Playwright version
Expand Down

0 comments on commit 8967b6a

Please sign in to comment.