This post is a companion blog to the earlier post on reuse. Here I will demonstrate how to generate scenario-based test vectors. Enjoy!
Month: July 2020
I think I’ve written this before: Revisiting Reuse
I’ve seen the question many times “Why do you care so much about reuse?” So giving my reusable answer I say “when you do something from scratch you have a new chance to make the same mistakes.” (1) If you look at your daily work, you will see we already reuse more than we realize.
When I go looking for images for “reuse” what I find most of the time are clever projects where you take a used plastic bottle and make it into a planter, or an egg carton becomes a place to start seeds.(2) What I want to talk about today is reuse for the same purpose, e.g. reuse like a hammer, a tool that you use to pound and shape the environment over and over.(3)
Hammer time(4)
Why do I care about reuse? Reuse is a company’s greatest asset; it is the accumulated knowledge over time. No one talks about “reusing a wheel” but that is what we are doing, we are reusing a highly successful concept.
So how do we get into the wheel house?(5) The first step is to identify a need, something that you (or ideally many people) need to use / do regularly.
When writing tests I frequently need to get my model to a given “state” before the test can begin. Creating the test vectors to do that manually is time consuming and error prone.
Once you have done that, think if there is a way that the task can be automated.
The solution I found was to leverage an existing tool, Simulink Design Verifier, and use the “objective” blocks to define my starting state. The tool then finds my initial test vectors to get me to where I want to be.
As described right now, this is a “proto-wheel.” It is a design process that I use (and have taught customers to use) but it is not fully reusable (yet). Why is that?(6)
Horseshoes and hand grenades(7)
This fails the “wheel test” in 2 fundemental ways
- It isn’t universal: every time I use it I need to recreate the interface, e.g. manually define the goal state
- It may not work: there are some conditions for which this approach will not find a solution.
Becoming a wheelwright: doing it right
If you want this to become “wheel like” you need to address the ways in which it fails. Here is how I plan to do that.
Create a specification language: by creating a specification language I, and my customers, will be able to quickly define the target state. Further, the specification language will ensure that errors in specification do not enter into the design
Analyze the design space: when a tool doesn’t work there is a reason; in some cases it can be deduced through mathematical analysis, in others, through analysis of failure cases. I am currently “tuning the spokes” on this wheel.
But will it roll? What is my (your) role (8) in making it happen?
In the end, a good idea and strong execution is not enough. The key to widespread reuse is getting it used by people outside the original (in this case testing) community. Until you do that it is only a specialty hammer.(9)
Getting those outside people to adopt a new tool or method is about getting people to care about the problem and the solution.(11)
Picking up a “hammer” is a 4 step process
- Know you have a problem: sometimes when you have been done something one way for a long time you don’t even realize there is an issue.
- Know there is a solution: if you don’t know hammers exist you will keep hitting things with rocks. It gets things done but your hand hurts.
- Have time to try out the solution: even the best hammer can be slower than your trusty rock the first few times you use it.
- Give the hammer maker time to make you a better hammer: chances are even the best tool will need refinement after the first few users.
Final comments: Why now?
Reuse reduces the introduction of errors into the system. Remember, “when you do something from scratch you have a new chance to make a the same mistake.”(13) And when working remotely during Covid, the chance to do so increases. Start looking at the tasks you do regularly and ask the need and automation questions.
Footnotes
- My wife Deborah always raves about my strawberry rhubarb crisps. But even after making over 100 of them over our 25 years together, I still can get the sugar to corn starch to lemon ratios a bit wrong if I don’t watch what I am doing.
- From a total energy usage I do question if we would be better off just recycling the bottle and egg carton.
- I started to think of the song “If I had a Hammer.” As it is a Pete Seeger song, it isn’t surprising that it is about civil and social rights. As a kid, the line “I’d hammer out love” seemed odd to me; hammers were blunt tools. When I got older I saw the other uses of hammers, to bend and shape things, to knock things into place. When you write software, write it like it’s a tool that can do all the things you need it to do.
- As a child of the 80’s “U Can’t Touch This” (hammer time) was at one time on nearly constant replay on the radio.
- First, make sure you are developing in an area that you know well so you know what needs to be done over and over.
- When I started this blog, I referenced the serial novels of the 19th Century. There are times I am tempted to end a blog post on a cliff hanger, but not today.
- When I first wrote the section title I thought “this must be a modern phrase” as while horse shoes have a long history, hand grenades are relatively new (e.g. 200 year old?) I was wrong. The earliest version can be traced back to Greek fire (or earlier)
- Homonyms, as a lover of puns, have always been something I have loved.
- If you thought I was done with the hammer metaphor you were wrong, I’m bringing it back at the end to drive my point home(11)
- Because that is what you do with hammers
- A blog post about this methodology would be one way of getting people to know about it.
- And in the spirit of reuse, I reused this from the start
Baskin Robbins: 31 flavors of models
Perhaps it is a coincidence, but, when I looked up the definition of “variant” online the example sentence was about an illness (1). The concept behind variants is appealing; within one model hierarchy, include multiple configurations of your target. But from a testing perspective, well….
If you go to Baskin Robbins ice cream, home of 31 flavors, and ordered a 2 scope cone you would have (31!/(31-2)!) 930 patterns, meaning it would take you almost 18 years to try them all if you did this once per week(2). So if you have an integration model with 8 referenced models, each model having 3 variants; well, just how many weeks do you want to test for?
Related and unrelated variants
To simplify the design and testing process the variants in a model should be related, e.g. if you have a variant for your car, manual or automatic transmission, then a related variant could be for the wave plate (auto) versus clutch (manual) models. An unrelated variant could be for the HVAC system.
Please note, the ice-cream references were supposed to end in the last paragraph, but the “which of these don’t belong” image I found had an ice-cream cone. That was not my intention but it is too late now.
Defining your inclusion matrix
What this matrix shows us (3) is which model variants
- Are allowed with other variants
- Are required by other variants
- Are not impacted by other variants
Validate the matrix
There are two primary methods for validating the inclusion matrix: inlined and pre-computed. With the inlined version, the full set of conditions for a variant to be selected is coded into the variant selection process. With the pre-computed version, the variant logic exists external to the component and the only final value is evaluated in the component.
As you would expect, there are pros and cons to both approaches (4). Having the logic in the component makes it easier for the developer of the module to understand what is going on; however it makes it more difficult for a system developer to have a global view of the variants. On balance the external computation is more likely to provide robust process.
Variants versus reuse
In our first example, an automatic versus manual transmission, this was clearly a variant; the models would be significantly different. But what if we were talking about a 4 speed versus a 5 speed transmission? Could that model, with some refactoring and change of data, be reused?
The key thing to keep in mind is that while a re-used model will require additional tests cases, many of the “basic” tests would already exist and could be modified for the reused data. On the other hand, a model variant will require a whole (or nearly whole) new set of tests.
Why now?
Why is this blog post part of the “impact of COVID” collection, as this is good advice at anytime. The answer is simple: as we social distance, the informal communication that makes poor design (5) tolerable is less prevalent.
This example COVID drives
Adopt this and thrive. (6)
Footnote
- The jest here is that the over use of variants can lead to multiple bugs in your software due to the increased complexity
- In reality this would take far less time as there are some combinations that should never be considered; for instance, anything with coconut.
- The Matrix (the movie) also shows us that Hollywood had (and has) an ill informed conception of what computer programming / hacking really looks like.
- All design is about the pros and cons, there are few things that are 100% “good”
- Informal communication makes that poor design tolerable, but it doesn’t mean that anyone is happy.
- I’m not sure if this is an example of “doggerel” or “bearerel” poetry. Hopefully you can bear with me.
Testing your testing infrustructure
Ah tests! Those silent protectors of developments integrity, always watching over us on the great continuous integration (CI) system in the clouds. Praise be to them and the eternal vigilance they provide; except… What happens to your test case if your test infrastructure is incorrect?
Quis custodiet ipsos custodes(1)
There are 4 ways in which testing infrastructure can fail, from best to worst
- Crashing: This is the best way your test infrastructure can fail. If this happens the test ends and you know it didn’t work.
- False failure: In this case, the developer will be sent a message saying “fix X”. The developer will look into it and say “your infrastructure is broken.”(2)
- Hanging: In this case the test never completes; eventually this will be flagged and you will get to the root of the problem
- False pass: This is the bane of testing. The test passes so it is never checked out.
False passes
Prevention of false passes should be a primary objective in the creation of testing infrastructure; the question is “how do you do that?”
Design reviews are a critical part of preventing false passes. Remember, your testing infrastructure is one of the most heavily reused components you will ever create.
While not preventing false positives, adherence to standards and guidelines in the creation of test infrastructure will reduce common known problems and make it easier to review the object
There are 3 primary types of “self test”
- Golden data: the most common type of self test is to pass known data that either passes or fails the test. This shows if it is behaving as expected but can miss edge cases(3)
- Coverage testing: Use another tool to generate coverage tests. If this is done, then for each test vector provided by the tool provide the correct “pass or fail” result.
- Stress and concurrency testing: For software running in the cloud, verification that the fact that it is running in the cloud does not cause errors(4)
- Time: Please, don’t let this be the way you catch things… Eventually because other things fail, false positives are found through root cause analysis.
Final thoughts
In the same way that nobody(5) notices water works until they fail, it is common to ignore testing infrastructure. Having a dedicated team in support is critical to having a smooth development process.
Footnotes
- In this I think we all need to take a note from Sir Samuel Vimes and watch ourselves.
- There is an issue here; frequently developers will blame the infrastructure before checking out what they did. Over time the infrastructure developers “tune out” the development engineers.
- Sometimes the “edge cases” that golden data tests miss are mainstream but since they were not reported in the test specification document, they are overlooked by the infrastructure developers.
- The type of errors seen here are normally multiple data read / writes to the same variable or licensing issues with tools in use.
- And if you look at it with only one eye, failures will slip passed.