Skip to content

Commit

Permalink
feat(output): Rename 'Fail' to 'Expected Failures' in test output
Browse files Browse the repository at this point in the history
Background
==========

The fuzzy tester has a bit of added functionality not present in many
testing tools: a test can be expected to pass, or expected to fail.

This is because a geocoder is a complex machine and we can't always make
a certain test pass.

We often use this when creating
tests for new functionality we haven't yet implemented. Marking the
tests as passing then becomes a bit of a celebration of a job completed
:)

On the other hand it might be because there's a known regression that
we can't currently fix, and have deemed acceptable. Maybe it came along
with an improvement in another area that is considered "worth it", or we
just plain don't know how to fix it yet.

This leads to 4 possible outcomes for any individual test. We currently
refer to them as follows:

**Passed:** the test was expected to pass and it did pass
**Improvement:** the test was not expected to pass, but it passed (usually
as a result of a bug fix or new feature)
**Fail**: The test was expected to fail, and it failed
**Regression**: The test was expected to pass, but it failed. This is
usually a problem and we look at these closely

The Problem
===========

Even for those of us familiar with the fuzzy tester, remembering if it's
'regressions' or 'fails' that represent a _new_, unexpected failure, can
sometimes be a challenge.

When viewing test output directly, the difference in color can be
helpful, but depending on how the tests are run, colors in the output
might not be available.

What's changed
==============

This PR replaces 'Fails' with 'Expected Failures' so that it's very
clear which of the two types of failures are which.
  • Loading branch information
orangejulius committed Mar 15, 2021
1 parent b78f4ac commit cdede47
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion output_generators/autocomplete.js
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ function prettyPrintSuiteResults( suiteResults, config, testSuites ){
console.log( '\nAggregate test results'.blue );
console.log( 'Pass: ' + suiteResults.stats.pass.toString().green );
console.log( 'Improvements: ' + suiteResults.stats.improvement.toString().green );
console.log( 'Fail: ' + suiteResults.stats.fail.toString().yellow );
console.log( 'Expected Failures: ' + suiteResults.stats.fail.toString().yellow );
console.log( 'Placeholders: ' + suiteResults.stats.placeholder.toString().cyan );

var numRegressions = suiteResults.stats.regression;
Expand Down
2 changes: 1 addition & 1 deletion output_generators/terminal.js
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ function prettyPrintSuiteResults( suiteResults, config, testSuites ){
console.log( '\nAggregate test results'.blue );
console.log( 'Pass: ' + suiteResults.stats.pass.toString().green );
console.log( 'Improvements: ' + suiteResults.stats.improvement.toString().green);
console.log( 'Fail: ' + suiteResults.stats.fail.toString().yellow );
console.log( 'Expected Failures: ' + suiteResults.stats.fail.toString().yellow );
console.log( 'Placeholders: ' + suiteResults.stats.placeholder.toString().cyan );

var numRegressions = suiteResults.stats.regression;
Expand Down

0 comments on commit cdede47

Please sign in to comment.