Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Code Coverage documentation #7027

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ Each test suite generates a report in CircleCI. For a complete overview of testi

Our code coverage is generated during the runtime of our unit, e2e, and visual tests. The combination of those reports is published to [codecov.io](https://app.codecov.io/gh/nasa/openmct/)

For more on the specifics of our code coverage setup, [see](TESTING.md#code-coverage)

# Glossary

Certain terms are used throughout Open MCT with consistent meanings
Expand Down
40 changes: 34 additions & 6 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,42 @@ Documentation located [here](./e2e/README.md)

## Code Coverage

* 100% statement coverage is achievable and desirable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😅

It's up to the individual developer as to whether they want to add line coverage in the form of a unit test or e2e test.

Codecov.io will combine each of the above commands with [Codecov.io Flags](https://docs.codecov.com/docs/flags). Effectively, this allows us to combine multiple reports which are run at various stages of our CI Pipeline or run as part of a parallel process.
Line Code Coverage is generated by our unit tests and e2e tests, then combined by ([Codecov.io Flags](https://docs.codecov.com/docs/flags)), and finally reported in Github PRs by Codecov.io's PR Bot. This workflow gives a comprehensive (if flawed) view of line coverage.

This e2e coverage is combined with our unit test report to give a comprehensive (if flawed) view of line coverage.
### Karma-istanbul

Line coverage is generated by our `karma-coverage-istanbul-reporter` package as defined in our `karma.conf.js` file:

```js
coverageIstanbulReporter: {
fixWebpackSourcePaths: true,
skipFilesWithNoCoverage: true,
dir: 'coverage/unit', //Sets coverage file to be consumed by codecov.io
reports: ['lcovonly']
},
```

Once the file is generated, it can be published to codecov with

```json
"cov:unit:publish": "codecov --disable=gcov -f ./coverage/unit/lcov.info -F unit",
```

### e2e

The e2e line coverage is a bit more complex than the karma implementation.

This is the general sequence of events:
1. Each e2e suite will start the webpack.coverage.js config with the `npm run start:coverage` command which configures webpack with the `babel-plugin-istanbul` babel-loader to generate code coverage during e2e test execution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is valid markdown, but let's number these 1, 2, 3, 4.. etc. Just so that if someone is not reading this in a markdown viewer they don't get confused.

Copy link
Contributor Author

@unlikelyzero unlikelyzero Sep 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fight you over this... In the markdown linter pr

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I'm up for this fight hahahah

1. Each e2e shard will generate only a piece of the larger coverage suite. This is converted with `nyc` with the `npm run cov:e2e:report` script
1.a Most of the tests are run in the '@stable' configuration a focus on chrome/ubuntu at a single resolution. This coverage is published to codecov with `npm run cov:e2e:stable:publish`.
1.b Some of our coverage only appears when run against `@unstable` tests, persistent datastore (couchdb), non-ubuntu machines, and non-chrome browsers with the `npm run cov:e2e:full:publish` flag. Since this happens about once a day, we have leveraged codecov.io's carryforward flag to report on lines covered outside of each commit on an individual PR.

### Limitations in our code coverage reporting

Our code coverage implementation has two known limitations:
- [Variability and accuracy](https://github.com/nasa/openmct/issues/5811)
- [Vue instrumentation](https://github.com/nasa/openmct/issues/4973)
Our code coverage implementation has some known limitations:
- [Variability](https://github.com/nasa/openmct/issues/5811)
- [Accuracy](https://github.com/nasa/openmct/issues/7015)
- [Vue instrumentation gaps](https://github.com/nasa/openmct/issues/4973)