In the last blog I introduced the best practices for designing scenario based tests. Today I am going to cover the, non Herculean(1), task of generating test vectors.
The “giddy” set-UP
Starting off happily let’s consider 3 things; the unit under test, the test harness and the analysis method.
- Unit Under Test (UUT): The UUT is what you are testing. For the test to be valid, the unit must be fully encapsulated by the test harness. E.g. all inputs and outputs to the UUT come through the test harness.(2)
- The test harness: (3)Provides the interface to the UUT, providing the inputs reading/logging the outputs. Test harnesses can be black, white or grey box. Test harnesses can be dynamic or static.(4)
- Analysis method: Dynamic or static; how the results of the test execution are evaluated.
Not to put the cart before the horse but; we start with a test scenario. We need the test vectors. To have test vectors, we need a test harness. To have a test harness we need a well defined interface.(5)
Within the software testing domain (which includes MBD) a well defined interface means the following:
- All the inputs and outputs of the system are known: Normally this is through a function interface (in C) or the root level inputs / outputs in a model
- Type and timing are known: The execution rate (or trigger) for the UUT is known as are all of the data types and dimensions of the I/O.
Time to saddle up!
No more horsing around, once you have your interface designed, it is time to create your test harness. Given that we are working in the domain of Model-Based Design, the ideal objective is to automatically generate a test harness. (To all the neigh sayers out there)
Signal time!
There are four basic methods for creating signals
- Manually: Ah…good old fashioned hand crafted test vectors. These take the most time but is where we normally start.
- Automatically (general constraint): The next step up is to create test vectors using an auto generation tool. These tools generally allow for basic “types of tests” to be specified such as range, dead code, MCDC.
- Automatically (constraints specified): The final approach is to use a test vector generation tool and apply constraints to the test vectors.
- From device: Perhaps this is cheating, but a good percentage of input test vectors come from real world test data. They have all the pros and cons(6); noise and random data; they may not get what you are looking for but…
In this example we have the UUT and a “Test Assessment Block” as our method for imposing constraints. What we program into the Assessment Block is what we want to happen, not what we are checking against(7). For example, we could specify the input vectors for the WheelSpeed, WheelTqCmd and SlipRationDetected are at a given value and that the output vector is ABS_PWM . The automatic test vector generation would then create a set of tests that met that condition. You could then check for the cases where the ABS_Fault should be active.
COVID-19 Acceleration: issues with “from the device”
When you social distance from your co-workers you are, more often then not, social distancing from your physical hardware. This directly impacts the ability to gather “real world” test data. My prediction is that we will see 4 trends as a result.
- Greater use of existing real world data / public domain data sets: Lets be honest, there are times that data is gathered because it is easy to do so; go to the lab run the widget, collect the data and go. However there is, no doubt within your company and within government, and university data bases a wealth of existing data that will match what you need down to the 90% level
- Increased automation of test data collection: To some extent being in a lab or in a vehicle will always be required for collecting data, however many of the processes around setup, data collection and data transmission can be automated to reduce both the time on site and the frequency of the time on site.
- Improved physical models: I know what you are thinking, this is about collecting real world data! What sort of trick is this(8)! What I am suggesting is that collection of physical data will be prioritized for the creation of better physical models to reduce the net time in lab.
- In use collection: The next step will be the transmission of data from existing objects in the field back to the manufacture. The model “IC-2021” freezer in the field will, most likely, share 95% of the same hardware and software. This means you have a lab in the field.
All of these methods will be used going forward to supplement traditional real-world data collection methods. With the physical modeling approach I am going to dive into how to select data to collect to rapidly improve the models. With the “in the field” we will look take our first look at big data methods.
Final thoughts
Test vectors are just one part of the overall testing infrastructure; the necessary starting point. We are going to keep looking at all the points along the Verification and Validation process; both in depth and at the impact that COVID conditions continue to have.
Footnotes
- With the use of one last Greek hero of antiquity, I hope to build a metaphor for the 12 labors of Hercules as applied to testing (with far fewer labors)
- We will look at how large the UUT should be in another blog post. For now, we will give the ballpark that a UUT should be linked to 5 ~ 8 related requirements. Each requirement will have multiple tests associated with it.
- A good test harness should be like the harness for a horse, e.g. provides a secure connection to the horse (software) enabling it to run fully, have the minimum number of attachment points (e.g. don’t overload with test points) and connect without chaffing (crashing or changing the behavior of the code).
- A dynamic test harness has the test validation software as part of the test harness, e.g. the UUT is evaluated as the test is run. A static test harness simply logs the data for post processing.
- Step 1 is to swallow a fly, today you will learn why!
- Noise is, and is not a problem. Since it will exist in the real world you should welcome noise into your test cases since that is what you will find once you deploy your product once and for all.
- As an example of what we want to happen, we may want to get an dessert (objective) but do not want one with coconut flavor (test).
- Not a very good trick, and 8! is 40,320.