Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Fix our flaky test problems #4055

Open
unlikelyzero opened this issue Jul 26, 2021 · 0 comments
Open

[CI] Fix our flaky test problems #4055

unlikelyzero opened this issue Jul 26, 2021 · 0 comments
Labels
type:maintenance tests, chores, or project maintenance

Comments

@unlikelyzero
Copy link
Contributor

unlikelyzero commented Jul 26, 2021

We have quite a few instances of flaky tests affecting our PRs and causing unnecessary heartburn. This ticket captures some of the strategies available to be prioritized and executed during Build 5.

We have 3 categories of flaky tests:
A. Non-atomic. As of 07/26, when the tests are run in a random order, 3-8 tests fail.
B. Cross-talk or test isolation. When tests are run in a serially, certain tests fail 30% of the time. Possibly state leakage between tests or general memory leaks.
C. Browser/OS/Environment discrepancies. We're on rolling versions of node and chrome without a way to distinguish those changes vs legitimate PR failure.

Some solutions (more to come)

  1. Run the circleci jobs every night to generate a baseline and watch for browser-specific regressions. Addresses C.
  2. Hunt down memory leaks.
    1. Stackoverflow with a listing that plagued jasmine 1.x and 2.x.
    2. Worth checking this, too. Addresses B.
    3. Memory leak in Import/Export Plugin #4060
  3. Create a nightly job which executes the tests in a random order. Addresses A.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:maintenance tests, chores, or project maintenance
Projects
None yet
Development

No branches or pull requests

2 participants