Model-Based Design

I recently taught a class on testing fundamentals, in it I made the comment that by my estimate there are over 8,000 lines of code in MATLAB dedicated to the simple (simple seeming) test

A == B

Why?  What is testing for equality so hard?  Let’s break it down.

1. Data types and numerical precision: depending on the selected data type the resolution to determine “equal” may not be present.  You can end up with false positives and false negatives
2. Tolerances: You can take data type into account by adding tolerances into comparison.
   1. Absolute tolerance: abs(A-B) < tol
   2. Relative tolerance: abs(A-B) < per_tol * A
3. But what about zero: Percentage tolerance is good, but what do you do when the value is zero?
   1. Relative tolerance (mean): abs(A-B) < per_tol * mean(A);
   2. Realitive tolerance (max): abs(A-B) < per_tol * max(A);
   3. Realitive tolerance (moving average) : abs(A-B) < per_tol * mean(A(i-N) : A(i+N))
4. What about noise: for measured data how do you handle the “junk data”?
5. What about missing data: much like junk data what do you do with missing data points?
6. What about data shifts (temporal or other): it is fairly common for comparison operations to take place with “shifted” data.  Where one signal is offset by some fixed amount in time.
7. What about non-standard data formats: how do you handle the comparison of a structure of data?  Do all elements in the structure have to match to “pass”?   Do you apply the same standard of tolerances to all elements?

You can quickly see where my estimate of 8K lines of code come from.  Why then do I mention this?  Two reasons

1. Start thinking about the complexity in “simple” tests
2. Stop creating test operations when they already exist

Endnote

This is written in the context of testing.  Any sort of algorithmic or logical code will, of course, use comparison operations.  For those cases keep 2 simple rules in mind

1. Do not use floating point equivalence operations:
2. Take into account the “else” conditions

If you find this content useful, please consider subscribing

Recently I had a discussion about the meaning of words with my wife.  Most of the time, most people, play some degree of fast and loose with the definition of words and the structure of their sentences.  However, there are some aspects of life and work where that will not stand, doctors visits, political discussions, and the writing of requirements.  With that in mind, here is a couple simple “mad lib” for writing a clear requirement.

Form 1 Response Requirement: When <Subject> is <State> then <Action> shall happen to <Action object>.
In this form, the requirement specifies a response.  For example, when my wife (subject) comes into the room (state) then I (action object) smile (action).

(Note: this should be fleshed out with definitions of “the room” and “smile”; e..g how long after, for how long.  The good news is that this is a testable requirement.  My wife enters rooms all the time so I can test it out tonight!

Form 2 State Check: When <Subject> is <State> then <Measured Object> shall have value <State>

This form enforces existing conditions, it can also be written in a “Before <Subject> is <State>…” form.  An example is “Before the car <subject> is placed in park <state> the vehicle <measured object> shall have a velocity less then 0.1 mph <state>.

So what templates do you use?

 

 

Completion of a task is accomplished by performing a sequence of steps.  The more steps in the sequence the more likely you are to make a mistake; either by forgetting a step or doing the step out of order.  One method for reducing the likelihood of making a mistake is the creation of sub-tasks.  This is where the analogy to dancing comes in to play.

When you first learn to dance you learn basic steps; the waltz’s box step, the tangos 8-count “Slow, Slow, Quick Quick Slow”… Once the basic step is mastered (and heaven help me one day I will master the box step) then additional “sub-tasks” can be learned.  There are four virtues of sub-steps.

1. Low chance of order mistakes:  shorter tasks have a lower risk for errors due to their simplicity
2. Low cost for errors: if a mistake is made in a sub-task it is, often, isolated to that sub-task and it can be quickly re-run
3. Decomposition: frequently when broken into sub-tasks, the task can be distributed to multiple people.
4. Ability to chain together: The sub-tasks can be decomposed into multiple “routines” and reused in multiple processes.

In general, processes that have 3 ~ 5 steps are considered “easy” to remember and master.  Going above 8 steps in a process results in increased possibilities of human error.

 

 

In today’s post, I will address some commonly asked question:

How long does it take to learn MATLAB and Simulink?

The answer to this question is dependent on a number of factors.  First, do you already know a programing language?  Are you familiar with control algorithms?  Do you have supporting people who know the toolset already?

Assuming a basic level of programming & software knowledge and a controls background with basic support most people will start using Simulink to basic develop control models within 2 ~ 3 weeks.  Over the course of 3 ~ 4 months, they will learn how to develop more complex systems.  For most people that level of understanding is sufficient for all of their development needs.

Deeper master of the tools, as is required for people to develop a group’s modeling patterns and best practices can be learned over the course of 3 to 5 years.

What is test coverage and why is it important?

Test coverage is the measure of how well the software is tested.  This can apply to MCDC coverage (the if/else checking) range coverage (e.g. did you hit every point in your tables) and temporal coverage (do you account for temporal, e.g. integral, effects).  Test coverage then tells you if you are sufficiently exercising the code base.  One important thing to keep in mind, it is possible to have 100% coverage and still have incorrect behavior.

How do I transition from a C based to Model-Based Design environment?

The answer, in part, is dependent on the state of your existing code base.  If it is well organized (e.g.encapsulated with independent modules) then the process is straightforward, individual modules can be replaced or extended on an as-needed basis.

More often then not when the transition takes place it is happening because there the existing code base is “stale” or difficult to maintain.  In these cases a strategic decision needs to be made, what part of the code base can be “walled off” or trusted while you work on new critical systems?  Once that decision is made the work begins in earnest.  Ideally, the models are derived from the base requirements, not engineered from the existing flawed code base.  Often this is when the lack of the original requirements are uncovered.

 

It is a common sense statement that you want your software to pass your validation tests.  However, if you always pass your tests how do you know your test can catch a failure?

Step 1: Validating tests components

Ideally, when you write tests you are assembling them from validated components, e.g. every time you write a test you are not reinventing the wheel.  For a test component to be validated the following must hold

• Fully defined interface: The inputs and output(s) must be fully specified, e.g. if the test only works for integer data that must be specified.
• Invalid data handling: The test component must respond correctly to invalid data, e.g. provide a failure message.
• Validated test cases (positive, negative, invalid data): The test component should have test cases which show the response to data that results in a pass, failure, and response to invalid data.

Step 2: Validate the order of execution

When tests are created there are a series of steps taken to execute the test

1. Environment set up: In this stage, the test environment is instantiated, running a test in the wrong environment will invalidate the test (for example running a fixed point test in a floating point environment)
2. Test set up: The Unit Under Test (UUT) is loaded and the testing tools are brought online.
3. Stimulus: This is the test execution, data is fed into the system and the outputs are monitored
   1. Analysis: In a subset of cases the evaluation takes place during execution.
4. Evaluation: Now the data collected during execution is put through the test components to determine the pass / fail status.
5. Cleanup: While not explicitly part of the test the clean up gets the environment ready for the next test…

Failure to follow these steps can result in incomplete or invalid test results

Step 3: Validating the test

At it’s most basic testing is about stimulus and response; the unit under test (UUT) is provided a stimulus and the response of the unit is observed and compared to an expected behavior.  This is the “expected input / expected output” data most commonly discussed in testing.  In this context validating the test implies

• Exercising range of inputs: The test should exercise the unit across its’ full range of inputs and configurations.
• Exercise invalid inputs: Unless the component has input checking the unit should be validated against invalid inputs, e.g. what happens if you give an infinite velocity.
• Sanity check of outputs: When first developing the test visual inspection of the outputs provides a quick
• Inspecting the outputs: The outputs from the unit to the test component needs to be validated against the components expected input format.

Clearing up misconceptions

This blog is not saying that the UUT needs to “fail” the test to show that the testing is valid, rather it must be possible for the test component to return a failed condition.  Here are my 7 favorite “It never fails” examples

1. Checks for negative values… when the input is an unsigned integer
2. Looks for errors lasting more than 30 seconds… in a 10-second test
3. Check for a function call… when the parent function is never called
4. The difference less than X tolerance…. when the tolerance is an order of magnitude greater than the signal
5. Check for consecutive occurrences…. when the data sampled is missing records
6. The test code is commented out… yes, that happens

I recently read an article about identifying programmers based on their programming “style“.  One thing that was interesting about the study sited in the article was that the algorithm for determining who wrote the code was more accurate for mature programmers in that they developed unique styles and were less likely to rely on pre-existing code “snippets”.

Time to reconsider

Starting any sort of work is a balance between applying known patterns and adopting new strategies.  Adopting new strategies, or at least considering them, takes time and therefore you only want to do so when warranted. The question is “How do I honestly evaluate if a new solution is required”.  I would put forth there are 4 questions that you need to ask yourself honestly

1. Is this a new type of problem: If you are working on your 101st iteration of the same type of problem chances are your validated solution can be applied.
2. Is it new in whole or in parts: If it is something new, start scoping how much of the problem is “new”
3. How well do I understand the new part: This is the hard part, do you understand, in outline at least, what is new in the system?  Until you can say “Yes” to functional understanding you should not move forward.
4. Can the new parts be treated in isolation: Once you understand what is new and what “known” you can determine the degree to which the new parts affect the behavior of the known parts.

Why ask What?

Life is to short to spend time reinventing the wheel for known problems, at the same time the quality of your work is dependent on the understanding of your problems.  By knowing when new solutions need to be applied you can minimize the amount of needless reinvention of the wheel.

The questions listed above are intended to spark a “is this a wheel” understanding.  What are your metacognition questions?

Every once in a while it is good to take a look outside and see what other people are doing, with that in mind here are a few blogs/articles that I have read recently

1. Continuous integration methods with Simulink projects
2. Execution order inspection for Simulink Functions
3. Using the Simulation test manager
4. Model-Based Design for Safety Critical Automotive systems
5. The challenges and benefits of Model-Based Testing
6. Model-Based Design for game architecture
7. Model-Based Design architecture for large systems

So tell me, what are some of the articles you have been reading?

Difference reports are a staple of software development, they allow you to quickly see what has changed.  Model-Based Design, with its graphical environment, presents some unique issues in differencing; we will examine them with this post.

Differencing best practices

The following best practice hold regardless of the development environment

1. Reasonable diffing cadence: Finding the difference between two objects is easy if the number of changes are small; however if large structural, or multiple-point changes have occurred difference operations become difficult if not impossible.  Perform differencing operations when no more then 5% of the object has changed.
2. Diff function changes / ignore visual changes: Visual changes, such as the use of indents in textual languages, or block placement in graphical, do not impact the behavior of the code.  While these changes may violate modeling guidelines they should be ignored during the diff operations
3. Include the authors: Have both the original author and updating person take part in the review for complex issues.
4. Test after diffing: After performing the difference operations have been run the files should be run through their test suites to ensure that the changes did not have a negative impact.

Don’t judge a model by its cover

In text based languages changes in the text are it; with Model-Based development environments it is the combination of the

• Block connections: How the blocks are connected to each other determines execution order and data flow.
• Block parameters: How the block is parameterized determines the functionality of the block
• Model configuration: The model can have parameters that influence the behavior system beyond the data set by the blocks

Final comment

It is possible to diff the underlying text files that define graphical models; at one point in time this was the only method that was possible.  However, doing this presents multiple problems and should be avoided.

1. Determining what is significant: The text representations often encode information into parameters / structures.  Often it is not obvious what are significant changes versus cosmetic.
2. Block changes: The text representation can have the same functionality “shifted” in the code to a new location depending on where it shows up in the model; however this is often not a functional change.
3. Interpretation: When reviewing the text changes the reviewers then have to interpret what the text means.

Table lookup algorithms provide a powerful tool for the modeling of systems the following are some basic tips for making your table look up’s fast and accurate

Pre-process your data

Table data often comes from real-world sources.  This presents two issues

1. Errors in the data: Validate that any anomalous readings are removed from the data
2. Non-uniform input axis: Real-world measurements frequently have jitter on the exact “place” of measurement.  Data should be renormalized to a uniform input axis

Know your boundaries

With tables, there are always limits on the input axes range.   Understanding the behavior of the table outside of the known range is important.  There are two basic options

1. Hold last value: In this instance, the last value at the tale boundary is used for all data outside of the axes
2. Interpolate data: In this case, the output data is extended beyond the known boundary.  For data near the boundaries this is generally acceptable; however, the accuracy can quickly break down as inputs exceed the

There are three approaches to handling this issue

1. Expand the range of valid data: This is the ideal solution but is often not possible due to sampling reasons.
2. Pre-interpolate the data: Create data outside of the range with “safe” values based on engineering knowledge.
3. Limit the input data range: Create a “hard stop” at the data edge.

Reduce, reuse, recycle…

It is common for multiple tables to share the same input axis; in this case, sharing the index lookup across multiple tables is one method for reducing the total number of calculations required by the algorithm.

Data types and discontinuities

Table

lookup algorithms are not well suited for data with discontinuities.  When working with such data either piecewise approaches are used or the region in which the discontinuity appears has additional data points to handle the sharp change.

A similar issue is working with integer-based data and interpolation.  When the outputs from the table are integer values then either interpolation should not be enabled or all possible input coordinates should be part of the input axes.

So there you go, a few quick suggestions for working with tables!