Last week the “Z” key popped off of my laptop. As my standard language of communication is English this has not been a big deal; however on Friday when I went to write to an email in German things changed. More correctly the task (typing) was the same but the mode (language) changed. Never one to let a random thing pass I thought “wow this is a great topic for a blog post”.(1)
Testing in the “comfort zone”
It is a well known problem with testing that we tend to test things that we know the best, as a result we end up with incomplete test coverage. There are two problems, first the obvious one that we are not covering everything. Second we are wasting time by creating redundant tests in the “well known” areas.(2)
How to exit your comfort zone?
Let’s look at three types of models, mode based, continuous (e.g. equation), logical and hybrid(3).
- Mode base: existing the “CZ” with mode based models is simple; validate that you are testing every mode in the model.
- It should be noted that with Mode base models there is often a temporal component that must be validated.
- Additionally the “mode to mode” paths should be validated as often initial conditions can be different depending on which mode you have transitioned from.
- Continuous (equation): these models can be fully tested by exercising the full range of all inputs. For example, let say you have two inputs U1 and U2 with ranges [0 to 10] and [-5 to 5]. The test vectors would cover a range [0,-5: 10,5]. There are a few considerations.
- Test spacing: depending on how sensitive the output is to changes in the input the “steps” in the coverage needs to be adjusted. E.g. could you “step” inputs of U1 in 0.1 or 0.5 increments?
- Exclusivity: in some instances inputs are related, e.g. the temperature of your car’s engine is never less than the outside temperature.(4) This can reduce the test range
- Temporal: another factor is “how long” are you at each data point?
- Logical: are similar to mode based testing, however they lack the state information that mode based testing implies. Like mode based testing this is validated by exercising each logical path in the model. Tools like Simulink Design Verifier can be used to generate these test vectors.
- Hybrid: E.g. 95% of all models. This is where design of experiments comes into play. For large systems it may not be practical to test “every point”. However that is not the goal of testing, the objective of testing is to test every operational behavior.
The upshot
Good tests take time; both to develop and to execute. Assuming a library of basic test functionality and a well written requirement document for the component you can estimate the number of test “points” as a function of the number of modes, inputs and logical branches.
TP = 1.25 * Modes + (min(0,numInputs/2) * numOutputs)^1.5 + numLogical/2; (5)
This formula empirical, and is derived from a review of test sets for well ordered models. The assumptions built into the formula are
- The “Mode-to-mode” connections are limited, e.g. not every mode can transition to every other mode
- There is a fair degree of mutual exclusivity in the input vectors.
- The number of tests is more sensitive to the number of outputs than number of inputs.
- Logical tests can often have redundant paths due to the lack of state information.
The final part of this equation is “time of construction”. Time of construction refers to how long it takes to create each test point. Both mode based and logic based tests vectors can be automatically generated, often achieving 100% coverage. (Or showing that 100% coverage is not possible and that there is an error in your logic). As a result I generally estimate the time to develop these tests as
t = (NumModes + NumLogic) * 1 minute;
The time assumes that some level of errors in modeling will be discovered and corrected. For the equation (continuous) testing the time is dependent on the coverage of the requirements; e.g. the more of the input space that the requirements cover the lower the total testing times.
t = Num_Req * 45 minutes + (10 minutes * %notCovered * 100)
Again, this is an empirical formula based on the well ordered models and an existing testing infrastructure.
Footnotes
- This blog post will be forwarded to my manager to explain why there is a $23 repair bill sent in
- I was once proudly told by an engineer they had over a 100 tests for their system; the problem was that those 100 tests were all dedicated to 6 out of the 27 functions in the system. We corrected that issue.
- That sentence should have been tested, as there are 4 “types” in there, not 3. This is what happens when you “design” a sentence with conceived notions.
- Ok, that statement isn’t absolutely true, if you had your car in a “cold box” and then drove it out into a warm day for a short period of time that would be true. At the same time if you are storing your care in a “cold box” you
- Hmmm, ending with a “;” shows just how long I have been programing in MATLAB.
- Again, thank you https://smbc-comics.com/
Please consider subscribing for regular updates