I come to you now with white sails unfurled. For the last two years I have walked the twists and turns of the Model-Based Design labyrinth, working in depth with a single customer. All the while marking the walls with chalk and unfurling my ball of twine so that having slayed the minotaur of process(1) I could return to recount my deeds(2).
Two thirds of my way into the maze a shift occurred, one that happened for all of us; the onset of the COVID-19 virus. Like an earthquake, the effects of the virus had an impact on the “shape” of the maze. Some passages had small changes, some massive deadfalls. Being deep in the heart of the maze when it happened has given me insights into how Model-Based Design has, and needs, to shift in response. (Changes which I have found to be both of use now and of long term benefit to the development cycle)
To sail beyond the sunset
Two years ago when I set out on this odyssey(3), I had laid out a sea chart to guide people through the boundary waters of Model-Based Design. These two years have given me a chance to see both the Kraken and the Treasure in the depths of that sea. Two years ago I wrote an introductory post on scenario based testing; in which I laid out the rationale for the testing and a basic methodology for developing these tests; let us get go deeper.
The adventure starts now: the scenario
A scenario based test(5) should be described in two parts
- What happens: (description) e.g. what are the steps that take you from point A to point B.
- What cannot happen: (prescribed) As you go from point A to point B what if C happens then your test fails.
A,B,C, simple as 1,2,3 right? Well yes, if it ever was just A,B and not C.
The high C’s
Lets continue thinking about a state machine where our objective is to get from State A to B. For many years I my observation has been that the average number of state transitions to get between to “states of interest” ranges between 6 to 8. If each state visited along the way has 2 exit points and if there are multiple ways to get from A to B then the total number of described transitions is on the order of 6 to 8 and each transition can have multiple required conditions.
The seven deadly sins(6,8)
Ok, not really 7 but…
- What is proscribed can change: The allowed behavior or event often changes between your current states(7). Often the proscribed value of a variable in your starting state is what you need to make a later transition.
- The mistake: Setting a test to monitor “if variable X = 1” you fail
- Recommendation: Evaluate the scope of each proscribed behavior and assign to only the active state. (Note this has the side benefit of faster running tests!)
- The mistake: Setting a test to monitor “if variable X = 1” you fail
- Synchronicity of event: A common conditional logic that you will encounter is “(input1 == 1) && (input2==0)”. On the surface this seems reasonable; but what if input 1 and input 2 are discreet events and only occur for one time step?
- The mistake: The test is written in such a way that you have input1 and input2 hitting the correct value at the same time, but often in the “real world” there is some “jitter” in the signals.
- Recommendation: If you have signal that have “jitter” consider having a temporary buffer variable that hold the value for a set number of cycles.
- Note from the “labyrinth”: these sort of “bugs” take a long time to track down since the “test” passed but the actual device failed.
- Recommendation: For all event based inputs add a notation to the Interface Control Document (ICD). This should be used as part of the test structure to determine if you have accounted for jitter.
- The mistake: The test is written in such a way that you have input1 and input2 hitting the correct value at the same time, but often in the “real world” there is some “jitter” in the signals.
- Not an exact match: For floating point numbers the scenario detection needs to use “fuzzy logic”. E.g. if the scenario calls for the vehicle to accelerate to 88 kph then the test should be read “when VehSpeed >= 88 && VehSpeed <= 88 + delta”. E.g. give some “wiggle room” in the event.
- The mistake: This is a common violation, so much so that we have MISRA 13.3 to cover it.
- Recommendations: write “check checkers” to check your checks to validate they are valid(9)
- Notes from the “Labyrinth”: Surprising to me this error has been more common in test infrastructure than in the units under test. From what I have seen this is a function of not running the same analysis tests on the test code as on the production code. Remember, it dose not matter where the bug comes in)
- The mistake: This is a common violation, so much so that we have MISRA 13.3 to cover it.
- Not all routes are the same: This is an issue of under specified use cases; in the one case we can think of the difference of getting from the ground floor to the top floor of a 10 story building; option 1 take the stairs, option 2 take an elevator; both gets you there but one is better for your heart.
- The mistake: When there are multiple routes a “shake up” of the multiple route can occur where parts of one route are melded onto another
- Recommendation: For multi-route systems create one test case for each route.
- Notes from the “Labyrinth”: The existence of multiple routes is, often, not taken into account in the design specification. The multiple routes are found when analysis tools (Such as Simulink Design Verifier) are used.
- The mistake: When there are multiple routes a “shake up” of the multiple route can occur where parts of one route are melded onto another
Next Steps
Having covered the specification portion of the test case in the next blog post we will cover the best practices for generating the test vectors while reducing the human input. In this section we will cover how to do this and why they are different when working in the post COVID-19 world.
Footnotes
- It is the minotaur of process since process is about the journey between two, or more, states. Having successfully navigate the maze once future trips through are no longer mysteries.
- In reality I will be recounting both the deeds of my own work and the many wonderful people I have been working with over the past 2 years.
- If learning about Model-Based Design can be thought of as a hero’s journey then let me be your wise elderly mentor(4)
- Except, please, I don’t want to be the mentor who is killed off in the 4th act
- Scenario based tests can be derived from use cases.
- Branching out from Greek myths to medieval concepts.
- Living in California now I think complaining about the weather should be proscribe behavior. Even after two years we can’t get over how wonderful it is year round.
- Ok, to many bullet points may be one of the “new” deadly sins
- When I worked on the original version of the Model Advisor for The MathWorks we had more discussion about “what is a check (guideline) versus check (test function)” then I can possibly remember.