-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Code Coverage documentation #7027
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -37,14 +37,42 @@ Documentation located [here](./e2e/README.md) | |
|
||
## Code Coverage | ||
|
||
* 100% statement coverage is achievable and desirable. | ||
It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test. | ||
|
||
Codecov.io will combine each of the above commands with [Codecov.io Flags](https://docs.codecov.com/docs/flags). Effectively, this allows us to combine multiple reports which are run at various stages of our CI Pipeline or run as part of a parallel process. | ||
Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in Github PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage. | ||
|
||
This e2e coverage is combined with our unit test report to give a comprehensive (if flawed) view of line coverage. | ||
### Karma-istanbul | ||
|
||
Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file: | ||
|
||
```js | ||
coverageIstanbulReporter: { | ||
fixWebpackSourcePaths: true, | ||
skipFilesWithNoCoverage: true, | ||
dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io | ||
reports: ['lcovonly'] | ||
}, | ||
``` | ||
|
||
Once the file is generated, it can be published to codecov with | ||
|
||
```json | ||
"cov:unit:publish": "codecov --disable=gcov -f ./coverage/unit/lcov.info -F unit", | ||
``` | ||
|
||
### e2e | ||
|
||
The e2e line coverage is a bit more complex than the karma implementation. | ||
|
||
This is the general sequence of events: | ||
1. Each e2e suite will start the webpack.coverage.js config with the `npm run start:coverage` command which configures webpack with the `babel-plugin-istanbul` babel-loader to generate code coverage during e2e test execution. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I know this is valid markdown, but let's number these 1, 2, 3, 4.. etc. Just so that if someone is not reading this in a markdown viewer they don't get confused. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I will fight you over this... In the markdown linter pr There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure I'm up for this fight hahahah |
||
1. Each e2e shard will generate only a piece of the larger coverage suite. This is converted with `nyc` with the `npm run cov:e2e:report` script | ||
1.a Most of the tests are run in the '@stable' configuration a focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`. | ||
1.b Some of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR. | ||
|
||
### Limitations in our code coverage reporting | ||
|
||
Our code coverage implementation has two known limitations: | ||
- [Variability and accuracy](https://github.com/nasa/openmct/issues/5811) | ||
- [Vue instrumentation](https://github.com/nasa/openmct/issues/4973) | ||
Our code coverage implementation has some known limitations: | ||
- [Variability](https://github.com/nasa/openmct/issues/5811) | ||
- [Accuracy](https://github.com/nasa/openmct/issues/7015) | ||
- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😅