Why testing fails…

ForImage result for improper complex systems, the goal of “complete” testing can be impossible to reach.  This often boils down to issues of time and complexity.  This is the most “common” type of failure.  A second, more insidious type of failure is improper testing.

What is improper testing?

Improper testing arises from not fully understanding what is being tested or how the testing algorithms work.

Let’s take a simple example, response time to a step input.  For this example, we will use the following test requirementImage result for step input response

TR_1028: Within 0.5 seconds of the unit step input the control system will settle to within 5% of the input value.

Failure type 1:  Single point pass:

There are two errors with this test:

  1. The hard comparison for the driver input value.  A correct method for evaluating this would be
    abs(driver-target) < eps(target)
  2. The pass condition is a single point in time.  What this means is that the signal could continue past the target not settling.

stepMessureError

Failure type 2: “Noise” intolerant

To fix this error the settling time needs to be defined.  To fix this a new state of “Settling” is added.   However, we have now introduced a  new error

  1. Hard failure condition: the response two the signal can include overshoot before the signal dampens down.  The current test fails if the signal falls out of the “less then 5%” range at any time.

failure_1_p2

Solution to this issues: 

The solution to this issue is to define a max settling time and to trigger a failure condition if the signal has not settled within the time.

failure_1_p3

Failure type 3: Sampling

A common failure mode, when dealing with large data relates to sampling.  Let’s take a look at an error flag example.

TR_2034: During the duration of operation the check engine flag will not be active.

It is common when logging data in a system to downsample the data; e.g. log every 10th or 100th data point. If the flag is intermitent it is possible that the flag would not be logged.

There are two solutions to this problem; first, the code could be refactored to include an “ever high” flag.  The second option would be to reconfigure the sampling rate for this specific test.

Failure type 4: Matching problems…

One Image result for matchescommon test type is baseline comparisons.  There are three types of problems that arise when comparing baseline data

  1. Failure to align/sampling:  baseline data is often taken from physical systems or prior simulations.  In this case, alignment of the data between the simulation and the test data is critical.  Likewise, if the sample rate of the baseline and the simulation are different then the alignment becomes more difficult.
  2. Poorly defined tolerances:  comparisons can be made against, absolute, relative and percentage error.  Further, the data type (integer, enum, double) have an effect on the type of comparison that should be performed
  3. Scaling/Units: The final, common, baseline data issue is a failure to correctly account for differences in units and scaling between the to sets of data.

Summary

The types of failures described here are a subset of improper testing issues.  They represent a failure to understand the full scope of the behavior of the system or a failure to understand how the collected data can be inspected.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s