For complex systems, the goal of “complete” testing can be impossible to reach.  This often boils down to issues of time and complexity.  This is the most “common” type of failure.  A second, more insidious type of failure is improper testing.

What is improper testing?

Improper testing arises from not fully understanding what is being tested or how the testing algorithms work.

Let’s take a simple example, response time to a step input.  For this example, we will use the following test requirement

TR_1028: Within 0.5 seconds of the unit step input the control system will settle to within 5% of the input value.

Failure type 1:  Single point pass:

There are two errors with this test:

1. The hard comparison for the driver input value.  A correct method for evaluating this would be
   abs(driver-target) < eps(target)
2. The pass condition is a single point in time.  What this means is that the signal could continue past the target not settling.

Failure type 2: “Noise” intolerant

To fix this error the settling time needs to be defined.  To fix this a new state of “Settling” is added.   However, we have now introduced a  new error

1. Hard failure condition: the response two the signal can include overshoot before the signal dampens down.  The current test fails if the signal falls out of the “less then 5%” range at any time.

Solution to this issues: 

The solution to this issue is to define a max settling time and to trigger a failure condition if the signal has not settled within the time.

Failure type 3: Sampling

A common failure mode, when dealing with large data relates to sampling.  Let’s take a look at an error flag example.

TR_2034: During the duration of operation the check engine flag will not be active.

It is common when logging data in a system to downsample the data; e.g. log every 10th or 100th data point. If the flag is intermitent it is possible that the flag would not be logged.

There are two solutions to this problem; first, the code could be refactored to include an “ever high” flag.  The second option would be to reconfigure the sampling rate for this specific test.

Failure type 4: Matching problems…

One common test type is baseline comparisons.  There are three types of problems that arise when comparing baseline data

1. Failure to align/sampling:  baseline data is often taken from physical systems or prior simulations.  In this case, alignment of the data between the simulation and the test data is critical.  Likewise, if the sample rate of the baseline and the simulation are different then the alignment becomes more difficult.
2. Poorly defined tolerances:  comparisons can be made against, absolute, relative and percentage error.  Further, the data type (integer, enum, double) have an effect on the type of comparison that should be performed
3. Scaling/Units: The final, common, baseline data issue is a failure to correctly account for differences in units and scaling between the to sets of data.

Summary

The types of failures described here are a subset of improper testing issues.  They represent a failure to understand the full scope of the behavior of the system or a failure to understand how the collected data can be inspected.