Model-Based Design and high integrity workflows

First off what qualifies as “High Integrity Software?”  The base “reference” document that I use is the “NIST Special Publication 500-204: High Integrity Software Standards and Guidelines”


Originally written to support the nuclear power industry it provides a valuable insight into what it means to be “safety critical”


In short, the software must function dependably (in a measurable and definable fashion) for critical functions.  Critical functions being defined as having failure modes that could cause serious injury, loss of life or property.

Model-Based Design and safety-critical software

When considering software design using MBD methodologies for safety-critical software everything starts with the requirements and the validation that those requirements are correctly implemented (this is true for all software).   I consider 4 primary factors

  1. Enhanced understanding of requirements
  2. Enhanced traceability
  3. Handoff error reduction
  4. Automated code generation

Enhanced understanding of requirements

Model-Based Design improves the understanding of requirements in 3 ways.  First, in general, models are easier to interpret than code.  Second, models allow you to easily simulate and visualize their behavior simplifying the understanding of the requirements.  Finally, the ability to link requirements to sections of a model and have those requirements show up in test results improves the chance that the requirements will be correctly implemented.  Image result for simulation animation stateflow

Enhanced traceability

Traceability refers to the act of following the implementation, modification, and validation of requirements.  Model-Based Design improves this process since a single model can be used as the design artifact at multiple stages in the development.  Meaning that once the link between the requirement and the model is made it is maintained.

Image result for requirements

Handoff error reduction

The droppedHandOffhandoff of software artifacts between people and roles (e.g. software developer to software integrator to software test engineer) is a well know point for the introduction of errors.  With Model-Based Design, the same model is used at each stage preventing hand-off errors.

Automated code generation

The use of automatically generated code prevents syntactical errors to which people are prone.  Many standards now allow you to claim credit for the use of auto code in the prevention of these errors.

Final thoughts

Developing safety critical systems for any industry requires following common best practices and established guidelines.  Following a Model-Based Design approach helps with the automation and validation of many of these steps while avoiding critical handoff errors.

Getting more out of your test reports

Did I pass or did I fail?  Yes or No?  What more do I need to know?  Putting aside the failure case, where knowing how you failed is important, let’s start by talking about what information you can know and why you would care about it.

Passing information

Image result for passing laneFirst, remember that there are multiple ways in which a test can “pass.”  Just like in school there can be A, B and C passing grades.  The way the grade is determined is, in part, related to the test type.

  • Coverage:  Pass is determined by reaching a given level of coverage.
  • Standards compliance: Passing is determined by being under a given level of errors and not violating any “critical” standards.
  • Baseline: Executing within a given fault tolerance.
  • Performance: Execution of the process under a maximum and average time

File churn

Another metric of interest to the system and testing engineers is the file “churn rate”.  From the testing perspective, there are two types of churn rate.  First how often is the file updated, second how often is the file referenced by updated files.Related image

Files with high “self-churn” are under development and, in general, should have test cases added as the development process matures.  Files with high “reference churn” are, in contrast, generally mature files that are referenced as utilities or as data.  These files should be “fully” locked down with test cases.

Failure is an option

Just like with passing there are multiple types of failures corresponding to the types of “passing.”  The main question is what sort of information do we bring out from the tests?

Image result for failure

  • Cause of failure:  There are 4 primary causes of failure
    • Test did not meet explicit criteria
    • Test failed to run (test harness bug)
    • Dependency failure (supporting files not present or failing their tests)
    • Performance failure

For each type of failure different “rich” information is desired.

Explicit criteria

For the explicit criteria case the cause of failure, as defined by the test, should be provided.  Any relevant plots, error diagnostics (e.g. line of code or block in model), as well as expected results, should be provided.

Failure to run

In a subset of cases, the failure will be in the testing infrastructure.  In this case, the location of the test infrastructure failure should be reported.  To prevent these types of errors when the testing infrastructure is developed test cases for it should be created.

Dependency failure

A dependency failure is a case of the “expected criteria” failure.  E.g. when working with a system of models one or more of the dependent models or bits of data has an error.  Dependency errors can occur in one of two ways.

  1. The dependent model changed and errors were introduced
  2. The interface between the parent and dependent model changed (in the parent) causing errors

If the error is of the first type then the report is the same as in the explicit error case.  For the second case, an interface report should be created detailing the change in the interface.

Infrastructure performance

The final note for this post is a few thoughts on the infrastructure performance.  Over time as additional tests are added to a system the total time to execute tests will increase.  Therefore monitoring both the execution time of individual tests as well as reusable test components is critical.

Image result for infrastructure failure

When profiling the performance of individual test components having “tests for the tests” is important as you want to make sure that when you improve the performance of a test component you do not change its behavior.


A few thoughts on test orginization…

How do you organize your bookshelf at home?  By the author (works well for fiction), by topic (generally good for non-fiction), by size (let’s face it shelf space can be an issue in home libraries.)  Any of these approaches work fine for small libraries, but when the total number of books starts getting large additional information is required to help organize your library.

Image result for library

Metadata and organization

If you have ever used Twitter (disclaimer: I never have) you have experienced metadata in the form of the #ILikeCats or #MyPoliticianIsGreatYoursIsTheSpawnOfSatan.  Metadata is a tag on an object that allows you to augment the information about the object.

Image result for propertiesImage result for metadata

Properties versus metadata

Metadata should not be confused with properties.  Properties are something inherent to the object.  If we extend our book metaphor then we would see

  1. Properties
    • Book title: Nine princes in amber
    • Author: Roger Zelazny
    •  Instance properties
      • Type: softcover
      • Condition: average
  2. Metadata
    • #Genre_Fantasy
    • #Multi_world
    • #Reread

Image result for nine princes in amber

Properties and metadata for tests

Since properties and metadata allow you to organize tests how do I categorize them?  Equally important, what do I not include?

  1. Properties
    • Model name: Name of the unit under test
    • Test harness: Link to the test harness for the unit under test
    • Test name: Short descriptive name
    • Description: Longer descriptive text that summarizes the test
    • Requirement linkages: Links to any requirements covered by the test
    • Data: Input, and expected outputs
  2. Metadata
    • Supported projects:  To support model reuse the tests should tag for which project(s) the test is valid.
    • Supported targets: A list of the targets on which the test is supported.  Such as  SIL, HIL or PIL testing.
    • Test level: An indication of the frequency with which the test is run, e.g. check-in, nightly, weekly, build.  More complex tests that take longer to run should have a higher level.
  3. What not to include
    • Model and data dependencies: These dependencies can be programmatically determined.  Specification of these dependencies will, over time, become out of date.
    • Version history: The version history should be included in the version control software.

Why we care?

In the course of normal development, there is both a local testing step and a global testing step.  At the local level, developers run tests against their updated models.  However, the developers are not expected to know the full scope of the model they are working with; hence the use of metadata and test properties to allow the test environment to fully exercise the model once it is checked in.


By leveraging the test properties and metadata we can more easily reuse tests.  Absent the metadata tests would need to be duplicated across multiple test suites; duplication which can lead to the introduction of errors in the test environment.


The most challanging control problem: Home heating for multi-floor homes

Hot air rises; cold air settles.  This is a fundamental law of nature and yet most multi-level homes are not set up to handle this challenge (including mine).  Having recently tried a “smart thermostat” and been very disappointed in it’s performance I have started to conceptually design my own home solution.

Trial 1:  Zone alone

My first trial implementation was a simple manual “zone control” system; e.g. in the winter, the top floors are shut off so the heat rises, in summer the bottom is shut off.


This approach worked somewhat but since I have a three story house and the thermostat is on the middle floor I always had a temperature gradient that was greater then I liked.

Trial 2: Internet of things (IOT) and active zone control

We are now entering into the thought experiment part of the post.


In the diagram above S1 ~ S3 are simple temperature sensors that would connect over wifi to the master control (hence the IOT part of this project).  The “Master Controller” would have open and close the ducts for each zone and control the Fan and heating and cooling elements.

So far so standard…

What changes this from a standard system into an interesting problem (for me at least) are the optimization constraints that I am putting onto the system

OBJ 1: Minimize the heat differential across the floors
OBJ 2: Minimize the time to achieve target heat
OBJ 3: Minimize energy usage
OBJ 4: Prevent temperature over/undershoot (e.g. don’t let the top floor overheat in winter, don’t let the basement become an ice house in summer)

To meet these objectives I was going to need a plant model; one that would allow me to model

  1. Multiple zones
  2. Heat flow between the zones
  3. Heat input based on damper status
  4. Changing heat flux from the outside of the house (e.g. day/night cycles)

As often is the case I was able to start with a basic model from The MathWorks.

Winters bane: the cold

Let’s start with the winter time problem; heat rises in the house from the lowest floor out through the attic.  Fundamentally the equation can be expressed as

q = -k * Δu

Basic heat flow due to a  temperature differential.

The second set of equations, e.g. the cost function, I expressed in the following fashion

TotalCost = α1 * fobj_1 + α2 * fobj_2 + α3 * fobj_3 + α4 * fobj_4

For example, the first objective function can be written as

fobj_1= abs(T1 – T2) + abs(T2 – T3)

Setting the alpha weights

The cost function uses a set of α coefficients to set the weights for each cost function.  To set the value of those coefficients two things need to be understood

  1. The normalized value of the function:
  2. The “priority” of the objective:

If you told me the priority of the objectives were

α1== 4
α2== 3
α3== 1
α4== 1

That is not enough to set the alpha coefficients.  For example, in the first objective function the max value is roughly 20 degrees while the second objective has a max value of 600 seconds (the last two had values of 45 and 3)  Therefore my weighted objective functions become

α1== 4 * (600/20)
α2== 3 * (600/600)
α3== 1 * (600/45)
α4== 1 *(600/3)

Augmenting the model

The final step in this project, before running the simulations and optimizations, is to augment the model in support of the multiple zones.  Since the duct controllers do not have a position feedback I have only on/off control over each zone.  This was added to the model and the optimizations started.


The results

I modeled heat loss in each of the zones, with the maximum heat loss in Zone 3, the highest floor. The heat loss on each floor had an impact on the optimal control results the general pattern was same.


After the initial heating (note the basement started out colder) there is a repeated pattern in the vent control.  The basement, due to heat convection activates first; the top floor also activates.  The middle floor, due to the convection from the basement and lower heat loss then the top floor never activates.

Final thoughts

The model I developed for the house is based on many assumptions; the implementation of the control algorithm allows me to have optimal outcomes regardless of the actual heat convection and heat loss properties of my house.  If I am able to implement this in my house I will let you know how the model compared to the actual results.



Continuous integration testing and Model-Based Design

The practice of continuous integration (CI) is well established within traditional C development environments and is at the core of agile workflows.  Model-Based Design processes are able to extend the base level of CI testing through the use of simulation to validate functional behavior.  While this is possible in traditional environments it requires the development of specific test harnesses which the MBD environment provides for “free”

Continuous Integration: Background

There are 5 best practices for CI systems

  1. All source objects must exist in version control: Source objects are used to create derived objects such as C code or test results.  The derived objects do not need to be kept in the repository.
  2. All operations on the objects must be automated: Operations include code generation, running test, collecting code metrics.
  3. Any commitment to the repository should trigger a build and test operation:  The “level” of test can be set for each check-in.  If the impact of the check-in is localized then a subset of the tests should be run.
  4. Results from build and test are easy to interpret: Every build and test operation should generate a report.  Results in that report should be easy to interpret and, in the case of errors, easy to trace back to the root cause of the error (e.g. which file resulted in the failure.)
  5. Minimize build and test execution time: To the extent possible the execution time of the build and test operations should be minimized.

The continuous integration workflow

The continuous integration workflow has two primary paths, an event, and temporal based triggers.  With the temporally-based trigger, the testing environment is run at predetermined intervals, generally nightly and weekly test runs.  With the event-based triggers, the testing environment is “triggered” by the check-in of a model or related data objects.


Testing levels

To improve the speed of test execution the concept of test “levels” is introduced.  At each level, the pass/fail criteria are assessed and if passed the next level for that model can be run; failure results halts further testing.

Level Info Runs
Level 1 Smoke tests to ensure that the model updates and passes basic modeling guideline checks Check-in, Nightly, Weekly
Level 2 Functional testing against requirements Checkin, Weekly
Level 3 Coverage and functional testing Weekly

Final considerations

There are multiple continuous integration servers such as Jenkins and TeamCity.   Further, there are multiple version control programs that can be combined with the CI servers (such as Git or SVN).

The key consideration is the set up of testing infrastructure “patterns” that enable simple integration with the CI system down the road.  These testing patterns will be covered in a future post.

Managing company-wide rollouts

First a specification, with this post I am addressing the rollout of Model-Based Design in large corporations, e.g. companies with at least 5 divisions using the Model-Based Design methodologies.   Information on rollouts for smaller companies can be found in the post on “group roll out” in the Model-Based Design roadmap.

The discussion of deploying Model-Based Design across a company requires reviewing the working groups and roles as well as how to maintain common tools and methods across the company.

Roles within the company

There are three roles to address:

  1. Model-Based Design Working Group (MBD-WG)
  2. Model-Based Design Tool and Support Group (MBD-TSG)
  3. Model-Based Design Director (MBD-D)


Model-Based Design Working Group

The MBD-WG is composed of technical and process experts from each division.  In the initial company rollout, their role is to present their divisions best practices for MBD and, as a group, determine what the common best practices will be.  Once the initial process is delivered their role is to

  1. Suggest improvements to the process
  2. Identify issues with common process for their division
  3. Act as point of contact for their division on MBD


Model-Based Design Tools & Support Group

The MBD-TSG role is to

  1. Provide the common infrastructure for Model-Based Design
  2. Provide training on the use of the Model-Based Design infrastructure
  3. Analyse changes to the MBD tool suite for desirable features
    (e.g. review new releases of tools for features that will provide additional or missing functionality)


Model-Based Design Director

The director role is unique to larger corporations.  download.jpgTheir role is to both arbitrate between groups and to set the overall vision of how Model-Based Design will be implemented within the company.  To that end, there are several requirements for the person filling the role, both technical and managerial aspects.

1.) Understanding of traditional software development processes and Model-Based Design processes

The most obvious is an understanding of the fundamentals of traditional software development and Model-Based Design.  In the role of Model-Based Design Director, they will be responsible for articulating the reasons for the transition to MBD.  This leads to the second requirement.

2.) Experience arbitrating between competing objectives

It is common for different divisions to have developed different approaches to Model-Based Design.  The role of MBD-D includes arbitration between the members of the Working Group to help develop the common infrastructure plan.  In order to do this, the director needs to become familiar with the process and requirements of each division.  This leads to the third requirement.

3.) Dedicated time to the Model-Based Design establishment effort

A common failure point in the establishment of a company-wide Model-Based Design process is having the director’s time split between multiple projects.  During the first three to five years of the establishment effort having the director focused on the establishment of the MBD process is critical.  This includes time to visit each division, meeting with the working group, attending conferences on Model-Based Design.

4.) Ability to articulate the Model-Based Design vision

The final role is to provide information to groups both inside and outside of the active divisions.  Changing processes requires consistent commitment and having a person at the “top” who can articulate a vision greatly enhances the chance that the adoption will succeed.

Final thoughts

The rollout across a large organization presents a number of unique challenges that smaller companies do not face.  These challenges derive from the unique methodologies that each division may have developed during their implementation process.  To minimize these issues the sooner a Model-Based Design Director and MBD-WG can be established the more effective the rollout will be.




What is the measure of a model?

In past posts, I have written about writing understandable models (Simulink and Stateflow.)  With this post, I want to address measures of clarity (or it’s inverse complexity.)  Note: in this post, I will be focusing specifically on Simulink metrics as contrasted with C based metrics.

Measurements in the C world

So what should be Baby angel sharks with ruler at Deep Sea World_900x583px.jpgmeasured and how do you evaluate the measurements?  In traditional C based development there multiple metrics such as…

  1. Lines of code (LOC):  A simple measure of overall project “size”.
    Note: A sub-metric is lines of code per function (LOC/Func)
  2. Coding standard compliance: A set of guidelines for how code should be formatted and structured (e.g. MISRA)
  3. Cyclomatic complexity: A measure of the number of logical paths through the program
  4. Depth of inheritance: A C++ measure of how deep the class definition extends to the “root” class.  Can be applied to function call depth as well.
  5. Reuse: The degree to which code is reused in the project.
    Note: a better measure is the degree of reuse across projects but this is more difficult to capture with automated tools.
  6. Coupling/Cohesion: Measures of the direct dependencies of modules on other modules.  Lose coupling supports modular programming
  7. Much more… : A list of some additional code metrics can be found here:  Software Metrics : Wikipedia


Model-Based Design metrics

Within the Model-Based Design world, there are both direct and analogous metrics of the C-based versions.

  1. Total block count (TBC): The total block count maps onto the LOC metric.  Likewise, a Blocks per Atomic Subsystem can be compared to the LOC/Function metric.
  2. Guideline compliance: Modeling guidelines, such as the MAAB, map on to C based guidelines.
  3. Model complexity: Maps onto cyclomatic complexity.  It should be noted that the model complexity and cyclomatic complexity of the generated code will be close but not exact.
  4. Subsystem/reference depth: A measure of how many layers of hierarchy exist in the model
  5. Reuse: The use of libraries and referenced models that can directly
  6. Coupling: Simulink models do not have an analogous metric for coupling.  By their nature, they are atomic units without coupling.
  7. Much more….

Evaluating measurements

There is no such thing as a perfect model or perfect bit of C code.  In the end, all of the metrics above are measured against pass/fail thresholds.  For example, common thresholds for the Model Metrics include

  1. Blocks per atomic subsystem < 50
  2. Guideline compliance > 90%
    Note: with some guidelines that must be passed regardless of percentage


Measuring the model

With Models, as with text-based development environments, there are a variety of tools for collecting metrics.  Within the Simulink environment, it is possible to write stand-alone scripts to automate this process or use the Model Metrics tool to automatically collect this information.

Plants resting on a table

In a previous post I wrote about fidelity in plant models.  For this post I want to focus, briefly, on three fundamental plant modeling tools that can be used to create simple and fast plant models.

  1. Table-lookup functions and adaptive tables
  2. Regression functions
  3. Transfer functions

Lookup tables

Lookup tables form the backbone download.jpgof many plant models.  From thermodynamic property lookups to engine torque and power curves they allow users to quickly access multi-input / single output data.

The efficiency and memory usage of tables can be be dramatically improved when the conditions of their use is well understood.  Common methods include

  1. Index search from last index:  In cases of slow moving data it is faster to start the indexing into the table by starting from the last time steps value.
  2. Reuse of index across multiple tables:  In many instances the same X or Y axis will be used across multiple tables.  The calculation cost for finding the index can be decreased through a pre-lookup function.
  3. Chose the correct interpolation function:  The efficiency of tables is dependent on the interpolation function.
  4. Pick and condition your data:  Frequently data from real world measurements is not evenly spaced.  By preconditioning the data (e.g. smoothing out data and providing even spacing for axis…)

Regression functions

Regression functions allow for modeling of non-linear multi-variable functions.  Two cautions regression functions.  First is in ensuring that data is valid throughout the input variable range and that the accuracy is sufficient at all points in the range.


The second caution is to validate that the regression equation does not become computational burdensome.

Transfer functions

The final “simple” plant model that I want to cover is the use of transfer functionsimages.png.  Transfer functions can be used to model a system when the mathematical equations describing the system are know and involve state (e.g. derivative) information.  (Note for multi-input, multi-output functions State Space equations can be used)

Transfer functions have the advantage over table look ups and regression functions that the represent the actual underlying physical system from a theoretically correct perspective.   In some instances the function may need to be “tuned” to take into accounts imperfections in real world systems.

Final thoughts

In the end there are many different methods for creating plant models.  Frequently the “correct” approach is to use a combination of “simple” and “advanced” modeling methods.  Take the time to think about what you require from your plant and the correct combinations can be selected.images.jpg


Interface control documents and Model-Based Design

An Interface Control Document (ICD) can be defined as

document that describes the interface(s) to a system or subsystem. It may describe the inputs and outputs of a single system or the interface between two systems or subsystems.

Within the traditional textual based development process, an ICD was either a text-based document or, a UML type diagram or a combination of the two.  Within the MBD development process, it is not entirely clear if additional supporting documentation is required or if the Model can serve as the ICD.

With most topics that I write about I have reached a firm conclusion on what is the accepted best practice.  In this area, I still have open questions.  In this post, I lay out the pros and cons for using models as ICDs.

Why models are sufficient:

The simplest argument as to why models are sufficient is that the models can be used in place of UML diagrams provided the interface has the sufficent markup.  For example in the image below the Simulink Interface View provides the data types and rates of all the inputs and outputs to the system.


When the model is part of a model hierarchy than the calling structure can be derived from the model.  (Simulink Model Dependency View)

Why models are lacking:

While the two views above are good, they lack information that is commonly found in ICD documents; the function interface (e.g. the C or C++ calling methods) and the data interface.  The models contain and use this information however they do not, natively, display the information.   Note: this is a limitation of UML diagrams as also.

The next issue with models as an ICD document is a question of “push and pull.”  By having the model as a development artifact and the ICD document you need to implement a change request process.

Lacking-Leaders-4-NEW_04.gifWhat can be done?

Use of automatic report generation can augment the information provided natively by the model.  Doing this could, in fact, generate a “standard” text-based ICD with the advantage being that the model would stay the single source of truth.

As with most issues in the development where there is not a native tool it is the implementation of a process that helps to bridge the gap.  All ready for text-based ICDs people have change request processes in place.  The question with an MBD approach is who implements the change at the model, and at the system level?

As always, please feel free to share your thoughts in the comment section.




Understanding the merge block

In Simulink, the merge block is documentation states

“The Merge block combines its inputs into a single output line
whose value at any time is equal to the most recently computed
output of its driving blocks.”

As clear as that statement is there are still questions about the behavior of the merge block.  This post attempts to clarify those questions.

Quiet night

The first and most common question is “what happens when none of my subsystems are enabled/triggered?”  In this example, we have 3 subsystems “A”, “B”, and “C” which are enabled when the driving signal is equal to their enumerated namesake.  The output from the subsystems is a simple enumerated constant equal to the subsystems name.  E.g. subsystem A outputs a value of “A”…


However, the way I have configured the driving signal it includes an enumerated value of “abcs.D” in addition to the A,B,C values.


In this example when the value of the driving signal is equal to abcs.D none of the subsystems are enabled.  In this case, the merge block simply outputs the last value input into the system.

Default merge subsystems

In the example above there is an unpredictable behavior due to the lack of a “default” subsystem.



The “Default” subsystem should execute every time step when the other systems are not running.  In this example, it is enforced through simple logical operations.

Multiple execution

In versions of Simulink from 15a and on (it could be earlier but that is the latest I had installed on my laptop) if you try and execute more then one subsystem during the same time step you will get an error message

merge_2In this case, I tried driving my two subsystems “LiveCat” and “DeadCat” with a “Schrodinger Waveform”™ vector.  When the vector was in the “Both” mode then both subsystems were active and the error message was triggered.