Did I pass or did I fail? Yes or No? What more do I need to know? Putting aside the failure case, where knowing how you failed is important, let’s start by talking about what information you can know and why you would care about it.
Passing information
First, remember that there are multiple ways in which a test can “pass.” Just like in school there can be A, B and C passing grades. The way the grade is determined is, in part, related to the test type.
- Coverage: Pass is determined by reaching a given level of coverage.
- Standards compliance: Passing is determined by being under a given level of errors and not violating any “critical” standards.
- Baseline: Executing within a given fault tolerance.
- Performance: Execution of the process under a maximum and average time
File churn
Another metric of interest to the system and testing engineers is the file “churn rate”. From the testing perspective, there are two types of churn rate. First how often is the file updated, second how often is the file referenced by updated files.
Files with high “self-churn” are under development and, in general, should have test cases added as the development process matures. Files with high “reference churn” are, in contrast, generally mature files that are referenced as utilities or as data. These files should be “fully” locked down with test cases.
Failure is an option
Just like with passing there are multiple types of failures corresponding to the types of “passing.” The main question is what sort of information do we bring out from the tests?
- Cause of failure: There are 4 primary causes of failure
- Test did not meet explicit criteria
- Test failed to run (test harness bug)
- Dependency failure (supporting files not present or failing their tests)
- Performance failure
For each type of failure different “rich” information is desired.
Explicit criteria
For the explicit criteria case the cause of failure, as defined by the test, should be provided. Any relevant plots, error diagnostics (e.g. line of code or block in model), as well as expected results, should be provided.
Failure to run
In a subset of cases, the failure will be in the testing infrastructure. In this case, the location of the test infrastructure failure should be reported. To prevent these types of errors when the testing infrastructure is developed test cases for it should be created.
Dependency failure
A dependency failure is a case of the “expected criteria” failure. E.g. when working with a system of models one or more of the dependent models or bits of data has an error. Dependency errors can occur in one of two ways.
- The dependent model changed and errors were introduced
- The interface between the parent and dependent model changed (in the parent) causing errors
If the error is of the first type then the report is the same as in the explicit error case. For the second case, an interface report should be created detailing the change in the interface.
Infrastructure performance
The final note for this post is a few thoughts on the infrastructure performance. Over time as additional tests are added to a system the total time to execute tests will increase. Therefore monitoring both the execution time of individual tests as well as reusable test components is critical.
When profiling the performance of individual test components having “tests for the tests” is important as you want to make sure that when you improve the performance of a test component you do not change its behavior.