You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have quite a few instances of flaky tests affecting our PRs and causing unnecessary heartburn. This ticket captures some of the strategies available to be prioritized and executed during Build 5.
We have 3 categories of flaky tests:
A. Non-atomic. As of 07/26, when the tests are run in a random order, 3-8 tests fail.
B. Cross-talk or test isolation. When tests are run in a serially, certain tests fail 30% of the time. Possibly state leakage between tests or general memory leaks.
C. Browser/OS/Environment discrepancies. We're on rolling versions of node and chrome without a way to distinguish those changes vs legitimate PR failure.
We have quite a few instances of flaky tests affecting our PRs and causing unnecessary heartburn. This ticket captures some of the strategies available to be prioritized and executed during Build 5.
We have 3 categories of flaky tests:
A. Non-atomic. As of 07/26, when the tests are run in a random order, 3-8 tests fail.
B. Cross-talk or test isolation. When tests are run in a serially, certain tests fail 30% of the time. Possibly state leakage between tests or general memory leaks.
C. Browser/OS/Environment discrepancies. We're on rolling versions of node and chrome without a way to distinguish those changes vs legitimate PR failure.
Some solutions (more to come)
The text was updated successfully, but these errors were encountered: