## Testing Requirements: Part 3

In part 3 of this series, we look at how and why we create test harnesses.

## Post credit thoughts

MathWorks provides stellar documentation on the creation of test harnesses; the key thoughts should be on why we create them (to avoid modifying the UUT) and how we call them (from test runners!)

## Types: Not just for fonts…

Typographically inclined people can debate for hours the importance of font selection in the presentation of a message; a proper font(1) sets the right type of message.(2) With programs, data type is used to specify range and resolution of the data as well as determining the total memory and speed of your program.

## I is for integer, at least in FORTRAN

In modern programing languages there are three primary data formats; integer, fixed point(3) and floating point. The range of integer data types is determined by the size of the data, e.g. 8, 16, 32 or 64 bit data types and the signed / unsigned nature.(4)

Floating point data represents information as a combination of a sign bit, an exponent component and a fractional component. This gives floating point data a greater range and resolution; however this can lead to issues when performing operations on data of significantly different orders of magnitude. For an overview of floating point data I would recommend this Wikipedia entry.(5)

## What is the “point”?

Memory usage is easy to understand,(6) using all 32 bit integers in your program will use twice the memory of 16 bit integers; the same idea applies to floating point data types. The trade-off then comes with accuracy and speed. Integer based arithmetic is computationally faster then floating point; likewise, 64 bit floating point takes longer then 32 bit. The objective should be to use the smallest data type that fully represents your data after completion of the operation.

Tips and tricks!

• Overflow, underflow and precision loss can be detected through the use of simulation or tools such as Polyspace
• Disable range checking and overflow options for generated code to create more efficient implementations

## Footnotes

1. Here in this blog I lack the control of what font you see in your browser or in your email; for all I know you are using Wingdings (ðŸ•ˆï¸Žâ™“ï¸Žâ– ï¸Žâ™‘ï¸Žâ™Žï¸Žâ™“ï¸Žâ– ï¸Žâ™‘ï¸Žâ¬§ï¸Ž).
2. And of course typesetting is part of the act of printing, the precursor to publishing electronically.
3. Fixed-point could be considered a special case of integer data type, we will look at it in a future blog post.
4. E.g. an 8-bit unsigned integer would have a range of 0 to 2^8 (256), while a signed 16-bit integer would have a range of (2^15 – 1) to 2^15 (-32767 to 32768). In every case they have a resolution of 1.
5. There is a conceptual overlap between fixed point data floating point operations work; this link provides a general overview of fixed-point data.
6. There is a caveat here; all hardware has a “smallest data type.” If you specify a data type smaller then the smallest data type you do not see any memory savings.

## Testing requirements: Part 2

In this final video we are going to tackle the “Must never” and “Must always” cases for requirements. To do so, the use of mathematical analysis is employed to determine if the conditions hold.

## Afterword

In some instances “never” and “always” cases cannot be definitively proven; when this happens there are three alternatives. First, subdivide the model into a smaller unit under test and attempt to prove the condition at that level. Second, determine under what conditions the requirement cannot be proven and then implement guards against those conditions. Or finally, explicitly guard against the condition in your model.

## Port in a storm

“It is not enough to know how; it’s the when that is key.” With all projects, at some point in the development cycle problems arise.(1) When they do, questions of refactoring or porting should land on your shores. Often simple bug fixes or in-language refactoring is enough to resolve the issue; however when the impact crosses a critical threshold the answer is to port.

Port:
noun: a town or city with a harbor where ships load or unload.
verb: to transfer between two locations or environments

## When it’s not right: port(2)

Porting software can be a major endeavor. Prior to this action a 5 part assessment should be made:

1. Will this software module be in use for the long term(3) (e.g., will it exist in code deployed in the field; what is the life span of that product?)
2. What is the criticality of the error? Is it: life threatening, product unusable, product behavior degraded, or is it a minor usability issue?
3. What are the other issues facing the product and their criticality?
4. If fixed now in-language will a later porting be required?

Assuming that the metrics for 1 ~ 4 are high enough, and assuming that there is enough time before your deadline, it is time to port.

## Estimating port fees

How do you estimate the time it will take to port your module, e.g. what is your “port fee?”(4) Ask yourself the following questions:

• How long did it take the first time around; of that time how much was spent on determining what to do versus implementation?
• Is the new language better suited for the implementation?
• Is there existing testing infrastructure and can it be repurposed for the new environment?
• Is the module unique or similar to existing code in the new environment; e.g. how well do you know your new environment?

The last question, how well do you know your environment is key. If you are just starting out in a new environment, the recommendation is to learn on non-critical applications. Learning an environment should never be a high stakes game.

## Footnotes

1. Problems can
• arise: e.g. float up from the buried issues in a poor design
• fall: e.g. come from outside sources as new information or requirements are given
• saunter in from the side: e.g. issues of integration with teammates
All in all, problems are three dimensional in nature
2. Assuming you are moving forward, port is left.
3. Stones from pyramids and other monuments have over time found their way into other construction. It is possible that stones from the pictured are part of someone’s house. When determining if a software module will be in use “for the long haul” it is not enough to look at the full systems but all the stones that make it up.
4. While free trade ports are a hot topic in economics, they do not exist in software.

## Testing Requirements Part 1

This is part 4 in my video series on Model-Based Design testing. Today I look at how to test requirements of the form “When X happens the response is Y.” Follow these steps and then you too can enjoy testing.

## Post movie credits…

In the video I state there are two basic types of requirements, when X happens Y occurs and X never occurs. In the final video in this series we are going to look at how we can prove the “never occurs” case.

## Hindsight is 20/20: Looking back and forward

Most years the “year in review post” seems like a good time for a lighthearted review of what has been and what is to come; with the events of this year some amount of gravity seems in order.

## A safety critical life

With one of my my earliest mentee I described the job of an engineer as “trying to find the minimum in Murphy’s equation and its first derivative.”; that is to say, design for the best outcomes and protect against the worst. The blog posts this year have focused heavily on Verification and Validation methodologies (finding the minimum) and writing requirements (protecting against the worst). It is easy to see the parallels between these tasks for Model-Based Design and for life during Covid times.

## The pendulum swings

With the introduction of COVID-19 vaccines there is an understandable desire to “go back to normal”; in fact some studies suggest that post events we will swing in the opposite direction; depending on how you look at it, this could be seen as embracing life or throwing caution to the wind. They key as I see it is to embrace life while remembering the lessons learned as the pendulum will swing back.

## Looking forward 2021

If hindsight is 20/20, and yes I have been waiting all year to make that joke, as a ratio (20/21; 20/22;…20/99) our view into the future is less clear. In 2021 this blog will be moving to a bi-weekly update (Tuesday / Thursday) and I will be writing more “multi-part” posts to go into greater depth on topics of interest.

I want to wish everyone a safe and happy 2021; I look forward to your comments in the coming year.

Michael

## Engineering Process: Fault Lines (P3)

Welcome to the home stretch! In today’s post we finish up the engineering process for the fault insertion. We left the last post with three outstanding questions; in todays post, I will walk through the design decisions that I made in implementing this solution.

• How much of the tool should be automated?
• What is the end user responsible for implementing?
• How do we make this scalable?

## Automation and scalability

The weak point in this solution revolves around the naming and maintaining the fault injection names; e.g., as the end user adds new “faults” into the system there is a need to create and define the global data, ensure that it is unique, and add it into the test harness.

## Check and define

The first two tasks are accomplished using a masked subsystem and a setup script; when the user double clicks on the fault subsystem they are prompted for a fault name. Closing the dialog will then launch a script that checks and defines the fault data.

The first step is to check to see if the variable already exists; this is done by evaluating the base workspace to see if the data is already defined. If it is a warning, dialog is generated.

If the data does not exist, we assign the names of our fault data to the data store read and data store writes inside the library subsystem.

Finally, we assign the data to the base workspace. As part of this task we inspect the incoming signals to determine the data type and dimension.

## But wait!

This code takes care of what happens when you add a block. However, in the long term you are just opening up your model with the data already in place. The next step is to provide a method for saving off the data and adding it to the test harness. In the design stage I tried out methods for auto-triggering this save and add step; however all of them had potential data conflicts. In the end the solution was to provide a script that the user could run to generate the data for themselves.

## What remains?

At this point with this code, our end users have the basic functionality that is required. Additionally, because of how we implemented it, the fault blocks can be inserted anywhere in the model, not just between referenced models which was one of our “stretch goals.” What remains now is the “error checking” or ruggedization tasks required to have a product out in the field standing on it’s own. That, however, will be a future set of posts.

## Engineering Process: Fault Lines (P2)

Having ended the last post on a cliff hanger, I’m going to reward your return with an answer to two questions that no doubt plagued your thoughts. “Why Fault Lines?” and “what changed in the problem statement?” The first answer is because I am inserting a “fault” on the Simulink “line.”(1)

## Changes to the problem statement

At the end of the last post I had the problem statement

For a system of models, create a method for simulating dropped
signals for models not directly connected to the root level I/O

As I started working through possible solutions to this process I hit on a further problem. How do we keep this method from impacting the code generation process for the full system?

## The prototype solution

The normal method for creating tests that do not change the models’ released behavior is to create a test harness that “wraps” the unit under test. However, in this instance since our objective is to work with models inside the system this approach will not work. Fortunately Simulink provides utilities to enable this.

## Take control!

The Simulink Block “Environment Controller” is the first key; we can have a default path (e.g., what is deployed) and the fault path (e.g., what is simulated). When code is generated the blocks in the “faultStuff” subsystem are optimized out of the generated code.

## Command from the top?

Having solved our first (new) problem(2) how do we get the data from the driving test harness to the “faultStuff” subsystem? The solution, in this case, was to use global data.

While as a general practice the use of global data is not recommended, in this case it was considered an acceptable solution for 3 reasons:

1. The global data would only exist in the simulation version of the model
2. Creation of the global data would be automated and not available to the developer
3. Isolation of the global data could be validated as part of the fault-line tool

## Two new tasks and a way forward

In developing my rational in support of global data I created two new tasks for myself (reasons 2 & 3).(3) However, for the prototype I just needed to create the example version of the faultLine.

There are three parts to this solution

• The fault subsystem: this subsystem consists of two data store read blocks and one data store write block. Based on the value of “faultLineActive_<NAME>” either the model path data or the fault value (faultLineValue_<NAME>) will be passed onto the simulation.
• The data definition: to use the global data, the Data Store Variables must be declared as part of a data dictionary. The full definition of the data e.g., type and size must be provided.
• Excitation: the final part is the “driver” of the test; in this example a simple test sequence that turns on and off the fault at a set time.

## From prototype to release

Moving from a prototype to a released version of the tool asks a new set of questions.

• How much of the tool should be automated?
• What is the end user responsible for implementing?
• How do we make this scalable?

As with the last post, I will leave you on a cliff hanger…. What are the design decisions that take this from prototype to reality?

## Footnotes

1. Perhaps not as earthshaking as you hoped, but well, I do my best.
2. The process of creating a final released product is often one of iterative problem detection and solution, with each subsequent problem being smaller then the previous one.
3. And these new tasks fit in with the “iterative problem detection” description of footnote 2.

## Engineering Process: Fault Lines (P1)

Whether you prefer fishing(1) or gardening(2) metaphors, for the next few posts I’m going to take you into the process I follow when solving a problem to show you how I fish.

## The problem statement

As is often the case, I will pull a problem from work I have done across multiple customers. I do this for two reasons. First, I have vetted the solution “in the field,”(4) and second I know that this is an issue that is commonly faced.

How do we simulate dropped and corrupted signals inside our system?

Multiple

## The first step

The first step in designing a solution is to understand the problem. The questions that I have are…

• Does a solution already exist?
• What does “inside our system” mean?
• In the deployment code where and how can signals be corrupted?

The critical question to answer is “where and how?” There are three primary ways that signals and data can be corrupted. The first is through incorrectly written code; since there are other methods for determining this issue it is usually dropped from the analysis. The second method is through interrupt data events (e.g., the execution of a function is put on hold while a higher priority function executes). The value of data changes during the interrupt resulting in corrupted data. This is of potential interest, however it results in a very large testing scope as the interrupt can occur at any point in function. As a result solutions to this problem are placed at a lower priority.(5)

The final method is through corruption of transmitted signals (e.g., messages sent over CAN / LIN / RS-232).(6) Thinking about this issue, we get the solution to our second question: what does “inside the system” mean? Transmitted errors occur at the boundary of functions, and from there we have an answer to the first question: does a solution already exist…

For individual models we have a solution; test vectors can be created that simulate dropped and corrupted signals. However for a system of models, Simulink does not have a built-in method for dropping signals for models on the “inside” of the system.

## Problem Restatement

With this understanding I came up with a new problem statement. One that reflected the design I would build towards. As you will see in the upcoming blog, there is further iteration on the new problem statement as the life cycle issues are considered.(7)

For a system of models create a method for simulating dropped signals for models not directly connected to the root level I/O

## Footnotes

1. E.g. give a person a fish, they eat for a day, but teach a person to fish and they eat for life.
2. A teacher is a gardener who plants the seeds of learning in your mind(3)
3. That both of these metaphors are about foundational tasks of humanity gives an idea how old this concept is.
4. Since one of my customers was an agricultural company it was literally validated in multiple fields
5. That we talk about priorities with interrupts is only appropriate as the priorities determines if the interrupt can occur.
6. No interface, no matter how ancient ever fully goes away.
7. This is my “cliff hanger” to bring you back to the next post.

## Fault injection

Over the past few weeks I have been thinking about how to test systems while introducing faults at lower levels within the system. At the topmost level the problem is simple; faults can be treated as just a signal. At a lower level either the model would need to be instrumented or the computational chain would need to be understood.

## Where are the faults?

Within controls design a fault is defined as either an interruption in, or else a delay or “crossed” signal in the data flow. Our first question is to ask “where and how do faults occur?” Faults occur at the boundaries of systems, either though direct I/O or communication protocols (CAN/LIN/Ethernet…) or through errors in global memory (pointer corruption).

## Injection Blocks

In this example we wrap our UUT (unit-under-test) as a referenced model. In the test environment we point to a secondary referenced model that includes a fault-injection block. Taking this approach we can instrument the UUT without challenging the UUT.

The actual “injection” block can be comprised of almost anything. In this case I used simple test sequence block to short the signal after 5 seconds

## When to fault test?

Fault based testing is an edge case for testing; not all systems require or merit fault testing. A general rule of thumb is…

• If the system is safety critical
and
• The system directly communicates with external systems
and
• The system does not have redundant communication
then…

…the system should have fault testing implemented.