Short answer: the real world is noisy.  If you write tests that assume clean input data you are not exercising the system in a real environment.  So let us talk about noise.

Types of noise and sources of noise

For this article, I will define noise as signal data entering the system from the outside of the system.  Sources of noise include

1. Resolution limits: all measuring devices have a limit to their resolution.  A ruler with 1/8th-inch markings cannot accurately measure 1/16th-inch resolution.
2. External interference: Frequently there are secondary effects that change the measurement.  For example, when measuring a voltage it is common to have noise in the signal from other wires running nearby.  (Which is why for some sensitive measurements shielded cables are used)
3. Dynamic property: In some instances, the value of the property being measured is changing rapidly; any given measurement may be an outlier.
4. Human error: For devices that human operators, well we make mistakes in how we enter information…

Types of noise, generally, map on to the sources of noise.

1. Quantization (resolution): Characterized by “jumps” in the value.  In dynamic systems must be tolerant of the jumps.  For static (e.g. post-run analysis) the jumps can be “smoothed” using functions.
2. White (external): Characterized by random values around the “actual” signal.  Generally can be filtered using standard transfer functions.
3. Outlier (dynamic): Characterized by occasional values outside the trending values.  If the “standard” range is known then these outlier values can be ignored.
4. Systematic (human): Characterized by systems being executed in a non-standard order.  Systems need to be made recoverable from non-standard execution order.

 

Testing the wild-noise

The basic strategy for testing with noise is to “inject noise” into the system under test.  How we inject can again be mapped back to our 4 types of noise

1. Floor functions (quantization): Use a floor function to resolve signals to the nearest value of the inputs resolution.
2. White noise generator (white): White noise generators are common functions.  One important note, if the same “seed” is used for the white noise for all runs then this test has an inherent flaw.
3. White noise generator (outlier): There is a special case of the white noise generator where signals are more episodic and, generally, of a larger value.   In these cases, a statistical model of the outlier signals is helpful in creating this white noise generator.
4. Decision tree analysis (human): Creating test cases for human error can be the most difficult.  For state logic, it is possible to analyze the system to determine all possible paths.

In the end, including noise in your tests will result in more robust systems.