In part 3 of this series, we look at how and why we create test harnesses.
MathWorks provides stellar documentation on the creation of test harnesses; the key thoughts should be on why we create them (to avoid modifying the UUT) and how we call them (from test runners!)
Typographically inclined people can debate for hours the importance of font selection in the presentation of a message; a proper font(1) sets the right type of message.(2) With programs, data type is used to specify range and resolution of the data as well as determining the total memory and speed of your program.
In modern programing languages there are three primary data formats; integer, fixed point(3) and floating point. The range of integer data types is determined by the size of the data, e.g. 8, 16, 32 or 64 bit data types and the signed / unsigned nature.(4)
Floating point data represents information as a combination of a sign bit, an exponent component and a fractional component. This gives floating point data a greater range and resolution; however this can lead to issues when performing operations on data of significantly different orders of magnitude. For an overview of floating point data I would recommend this Wikipedia entry.(5)
Memory usage is easy to understand,(6) using all 32 bit integers in your program will use twice the memory of 16 bit integers; the same idea applies to floating point data types. The trade-off then comes with accuracy and speed. Integer based arithmetic is computationally faster then floating point; likewise, 64 bit floating point takes longer then 32 bit. The objective should be to use the smallest data type that fully represents your data after completion of the operation.
Tips and tricks!
In this final video we are going to tackle the “Must never” and “Must always” cases for requirements. To do so, the use of mathematical analysis is employed to determine if the conditions hold.
In some instances “never” and “always” cases cannot be definitively proven; when this happens there are three alternatives. First, subdivide the model into a smaller unit under test and attempt to prove the condition at that level. Second, determine under what conditions the requirement cannot be proven and then implement guards against those conditions. Or finally, explicitly guard against the condition in your model.
“It is not enough to know how; it’s the when that is key.” With all projects, at some point in the development cycle problems arise.(1) When they do, questions of refactoring or porting should land on your shores. Often simple bug fixes or in-language refactoring is enough to resolve the issue; however when the impact crosses a critical threshold the answer is to port.
Port:
noun: a town or city with a harbor where ships load or unload.
verb: to transfer between two locations or environments
Porting software can be a major endeavor. Prior to this action a 5 part assessment should be made:
Assuming that the metrics for 1 ~ 4 are high enough, and assuming that there is enough time before your deadline, it is time to port.
How do you estimate the time it will take to port your module, e.g. what is your “port fee?”(4) Ask yourself the following questions:
The last question, how well do you know your environment is key. If you are just starting out in a new environment, the recommendation is to learn on non-critical applications. Learning an environment should never be a high stakes game.
This is part 4 in my video series on Model-Based Design testing. Today I look at how to test requirements of the form “When X happens the response is Y.” Follow these steps and then you too can enjoy testing.
In the video I state there are two basic types of requirements, when X happens Y occurs and X never occurs. In the final video in this series we are going to look at how we can prove the “never occurs” case.
Most years the “year in review post” seems like a good time for a lighthearted review of what has been and what is to come; with the events of this year some amount of gravity seems in order.
With one of my my earliest mentee I described the job of an engineer as “trying to find the minimum in Murphy’s equation and its first derivative.”; that is to say, design for the best outcomes and protect against the worst. The blog posts this year have focused heavily on Verification and Validation methodologies (finding the minimum) and writing requirements (protecting against the worst). It is easy to see the parallels between these tasks for Model-Based Design and for life during Covid times.
With the introduction of COVID-19 vaccines there is an understandable desire to “go back to normal”; in fact some studies suggest that post events we will swing in the opposite direction; depending on how you look at it, this could be seen as embracing life or throwing caution to the wind. They key as I see it is to embrace life while remembering the lessons learned as the pendulum will swing back.
If hindsight is 20/20, and yes I have been waiting all year to make that joke, as a ratio (20/21; 20/22;…20/99) our view into the future is less clear. In 2021 this blog will be moving to a bi-weekly update (Tuesday / Thursday) and I will be writing more “multi-part” posts to go into greater depth on topics of interest.
I want to wish everyone a safe and happy 2021; I look forward to your comments in the coming year.
Michael
Welcome to the home stretch! In today’s post we finish up the engineering process for the fault insertion. We left the last post with three outstanding questions; in todays post, I will walk through the design decisions that I made in implementing this solution.
The weak point in this solution revolves around the naming and maintaining the fault injection names; e.g., as the end user adds new “faults” into the system there is a need to create and define the global data, ensure that it is unique, and add it into the test harness.
The first two tasks are accomplished using a masked subsystem and a setup script; when the user double clicks on the fault subsystem they are prompted for a fault name. Closing the dialog will then launch a script that checks and defines the fault data.
The first step is to check to see if the variable already exists; this is done by evaluating the base workspace to see if the data is already defined. If it is a warning, dialog is generated.
If the data does not exist, we assign the names of our fault data to the data store read and data store writes inside the library subsystem.
Finally, we assign the data to the base workspace. As part of this task we inspect the incoming signals to determine the data type and dimension.
This code takes care of what happens when you add a block. However, in the long term you are just opening up your model with the data already in place. The next step is to provide a method for saving off the data and adding it to the test harness. In the design stage I tried out methods for auto-triggering this save and add step; however all of them had potential data conflicts. In the end the solution was to provide a script that the user could run to generate the data for themselves.
At this point with this code, our end users have the basic functionality that is required. Additionally, because of how we implemented it, the fault blocks can be inserted anywhere in the model, not just between referenced models which was one of our “stretch goals.” What remains now is the “error checking” or ruggedization tasks required to have a product out in the field standing on it’s own. That, however, will be a future set of posts.
Having ended the last post on a cliff hanger, I’m going to reward your return with an answer to two questions that no doubt plagued your thoughts. “Why Fault Lines?” and “what changed in the problem statement?” The first answer is because I am inserting a “fault” on the Simulink “line.”(1)
At the end of the last post I had the problem statement
For a system of models, create a method for simulating dropped
signals for models not directly connected to the root level I/O
As I started working through possible solutions to this process I hit on a further problem. How do we keep this method from impacting the code generation process for the full system?
The normal method for creating tests that do not change the models’ released behavior is to create a test harness that “wraps” the unit under test. However, in this instance since our objective is to work with models inside the system this approach will not work. Fortunately Simulink provides utilities to enable this.
The Simulink Block “Environment Controller” is the first key; we can have a default path (e.g., what is deployed) and the fault path (e.g., what is simulated). When code is generated the blocks in the “faultStuff” subsystem are optimized out of the generated code.
Having solved our first (new) problem(2) how do we get the data from the driving test harness to the “faultStuff” subsystem? The solution, in this case, was to use global data.
While as a general practice the use of global data is not recommended, in this case it was considered an acceptable solution for 3 reasons:
In developing my rational in support of global data I created two new tasks for myself (reasons 2 & 3).(3) However, for the prototype I just needed to create the example version of the faultLine.
There are three parts to this solution
Moving from a prototype to a released version of the tool asks a new set of questions.
As with the last post, I will leave you on a cliff hanger…. What are the design decisions that take this from prototype to reality?
Whether you prefer fishing(1) or gardening(2) metaphors, for the next few posts I’m going to take you into the process I follow when solving a problem to show you how I fish.
As is often the case, I will pull a problem from work I have done across multiple customers. I do this for two reasons. First, I have vetted the solution “in the field,”(4) and second I know that this is an issue that is commonly faced.
How do we simulate dropped and corrupted signals inside our system?
Multiple
The first step in designing a solution is to understand the problem. The questions that I have are…
The critical question to answer is “where and how?” There are three primary ways that signals and data can be corrupted. The first is through incorrectly written code; since there are other methods for determining this issue it is usually dropped from the analysis. The second method is through interrupt data events (e.g., the execution of a function is put on hold while a higher priority function executes). The value of data changes during the interrupt resulting in corrupted data. This is of potential interest, however it results in a very large testing scope as the interrupt can occur at any point in function. As a result solutions to this problem are placed at a lower priority.(5)
The final method is through corruption of transmitted signals (e.g., messages sent over CAN / LIN / RS-232).(6) Thinking about this issue, we get the solution to our second question: what does “inside the system” mean? Transmitted errors occur at the boundary of functions, and from there we have an answer to the first question: does a solution already exist…
For individual models we have a solution; test vectors can be created that simulate dropped and corrupted signals. However for a system of models, Simulink does not have a built-in method for dropping signals for models on the “inside” of the system.
With this understanding I came up with a new problem statement. One that reflected the design I would build towards. As you will see in the upcoming blog, there is further iteration on the new problem statement as the life cycle issues are considered.(7)
For a system of models create a method for simulating dropped signals for models not directly connected to the root level I/O
Over the past few weeks I have been thinking about how to test systems while introducing faults at lower levels within the system. At the topmost level the problem is simple; faults can be treated as just a signal. At a lower level either the model would need to be instrumented or the computational chain would need to be understood.
Within controls design a fault is defined as either an interruption in, or else a delay or “crossed” signal in the data flow. Our first question is to ask “where and how do faults occur?” Faults occur at the boundaries of systems, either though direct I/O or communication protocols (CAN/LIN/Ethernet…) or through errors in global memory (pointer corruption).
In this example we wrap our UUT (unit-under-test) as a referenced model. In the test environment we point to a secondary referenced model that includes a fault-injection block. Taking this approach we can instrument the UUT without challenging the UUT.
The actual “injection” block can be comprised of almost anything. In this case I used simple test sequence block to short the signal after 5 seconds
Fault based testing is an edge case for testing; not all systems require or merit fault testing. A general rule of thumb is…
…the system should have fault testing implemented.
