Modular testing environments

Foundations define and limit the structures we create; this is as true in Model-Based Design as it is in architecture.  With that in mind, I want to use this post to discuss the concept of modular testing environments (MTE).  First, I will point to an earlier blog post “Testing is software“, before I drill deeper into the concept of MTE.

What is a modular testing environment?

A modular testing environment consists of 5 parts

  1. Test manager:test manager provides the framework for running, evaluating and reporting on one or more test cases. Further, the test manager provides a single hook for the automation process.
  2. Test harnesses: a test harness is the software construct that “wraps” the unit under test.  Ideally, the test harness does not change the unit under test in any fashion; e.g. it allows ‘black box’ testing.
  3. Evaluation primitives: the evaluation primitives are a set of routines that are commonly used to evaluate the results of the test.  Evaluation primitives range from a simple comparison against an expected value to complex evaluations of a sequence of events.
  4. Reporting: there are two types of reports, human and machine readable.  The human readable reports are used as part of the qualification and review process.  Machine-readable reports are used for tracking of data across the project development.
  5. Data management: testing requires multiple types of data, inputs, outputs, parameters and expected results.

Why is a modular testing environment important?

Having helped hundreds of customers develop testing environments the 5 most common issues that I have encountered are

  1. Reinventing the wheel, wrong:  Even the simplest evaluation primitive can have unexpected complexities.  When people rewrite the same evaluation multiple times mistakes are bound to occur.
  2. Tell me what happened:  When tests are pulled together in an individual fashion it is common for there to be limited or inconsistent reporting methods.
  3. Fragile tests: A fragile test is one where if the inputs change in a significant fashion the test has to be completely rewritten.
  4. “Bob” has left the company:  Often tests are written by an individual and when that person leaves the information required to maintain those tests leaves with them.
  5. It takes too much time:  When engineers have to build up tests from scratch, versus assembling from components, it does take more time to create a test.  Hence, tests are not written.

Final thoughts

Verification and validation activities are central to any software development project, Model-Based Design or otherwise.  The easier you make the system to use the more your developers will embrace them.

Managing data

In previous posts, I have covered data attributes and data usage.  In this post, I cover data management.  Within the Model-Based Design workflow, and traditional hand coding environments, there is a concept of model scoped and common data.  This blog post will use Simulink specific concepts for Data Dictionaries to show how scoped data can be achieved.

What is in common?

Deciding what goes118eb73e994c025de7f60b0689c4de10 into the common versus the model specific data dictionary is the primary question that needs to be asked at both the start of the project and throughout the model elaboration process.  There is always a temptation to “dump” data into the common data dictionary to “simplify” data access.  While in the short run it simplifies access, in the long run, doing so creates unmanageable data repositories.  So, again, the question is “what goes in there?”

Common data type specification

commonDataTypesThe common data types consist of four primary entries, each of which is created as a separate sub-dictionary.

  • Structure definitions
  • Enumerated data types
  • Data type aliases
  • Model configurations

In all 4 cases, these bits of information should be used in a global scope.  For example, structures used as an interface definition between two models or an enumerated data type that is used for modal control across multiple models.  In contrast, structures that are local to a single model should not be part of the common data types sub-dictionary.

Common data

Like the common data types, the commoncommonData data consists of sub-dictionaries.  In this case, there are three.

  • Physical constants
  • Conversion factors
  • Common parameters

The first two are simple to understand; instead of having the engineer put in 9.81 (m/s) for each instance of the force of acceleration a physical constant (accelGravMetric) can be defined.  Likewise, instead of hard coding 0.51444 you could have a parameter Knots_to_meter_p_sec.  (Note: in the first case, 9.81 is a value that most engineers would know off the top of their head.  The second case most people will not recognize and it results in “magic numbers” in the code.  This is compounded when people “compact” multiple conversion factors into a single conversion calculation and the information is lost)

The final sub-dictionary, common parameters, is the most difficult to scope.  Ideally, it should be limited to parameters that are used in more than one model; or more than one integration model.  To prevent the “mushroom growth” of data in the common parameter data dictionary regular pruning should be applied.

Pruning your data

Pruning data is the process of examining entries in a data dictionary and determining if they are needed in the common data or in a model specific dictionary.  Within the Simulink environment, this can be accomplished using the model explorer or programmatically


Model and integration model data dictionaries

In the section on model architecture, we discussed the concept of “integration models.”  An integration model consists of multiple sub-models, which, in turn, may contain sub-models.


The pattern for the integration model data dictionary mirrors the pattern that was shown in the initial diagram; the “twig” of the model tree references the branches, which in turn reference all the way back to the root.


Final thoughts

The use of scoped data dictionaries allows users to logically organize their data while minimizing the amount of work that individual contributors need to take to maintain the data.  This approach does not eliminate the need for data maintenance however it does provide tools to aid in the work.




Assessing your current state

During the exploration phase of adoption, the adoption team should have become familiar with the fundamentals of Model-Based Design e.g. model architecture, data dictionaries, testing, formal methods, report generation and so on. In that this blog is focused on groups adopting MBD it is a fair assumption that the current state is “introductory” (1) however the following sections can be used to identify where additional support is needed.(2)


The following tasks should be understood…


  • Encapsulation of models:  How to define each model in a format with defined interfaces and data.
  • Integration of models: How to integrate individual models into a larger integration model
  • Integration of existing code artifacts: How to integrate artifacts from the models into existing code artifacts and/or integrate code artifacts into the models

Data management

The following tasks should be understood…


  • Creation of data artifacts for use in the model: How to create data that the model can reference
  • Management of data artifacts: How to store and reference data artifacts in a scalable way.
  • Harmonizing data between model and existing code artifacts: How to reuse data between models and existing code artifacts


The following tasks should be understood…

  • Creation of test harnesses: How to create a test harness that will exercise the model in a “stand alone” method.
  • Creation of data: How to create the data used by the test harness both for input and output evaluation.
  • Creation of test evaluation methods: Creation of methods for evaluating the outputs from tests

Supporting tools

The following tasks should be understood…


  • Basic use of version control: The users should understand which files need to be placed under version control.

Final thoughts

The obvious question arises “how do I move from “introductory” to “ready?”  There are three primary methods

  1. Training: There are multiple training courses (industry) out there allowing you to learn about Model-Based Design (university).
  2. Papers: Both academic and industry papers exist to help you learn about MBD.
  3. Outside help:  Outside help can come from either hiring people with MBD experience or hiring outside consultants.
  4. All of the above…


(1) Having an introductory level of knowledge of Model-Based Design methodologies is not the same thing as an introductory level of knowledge of software development.  Most groups that adopt MBD have a strong software development background.
(2) Most of the “Why is my state so [XXX]” I understand.  However, I don’t know why Pennsilvina and Connecticut got “haunted” as their “Why.”

KPI for initial project…

What will and what were you measuring?

During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured.  However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).

First a word of warning, metrics are useful but they rarely provide the full picture(2).  That being said there are metrics that can be monitored

Bugs found at stage X…

One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process.bugs However, one side effect is the observation “we find so many bugs when following an MBD process.(3)”  Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process.  To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…

cost = Severity * StageConstant

Test and requirements coverage

Model-Based Design allows users to track the coveragePolyspacerequirements and testing coverage through automated tools(4).  With respect to test coverage; there are two types of test coverage.  The first is requirements based test coverage; do tests exist to cover the requirements.  The second are formal metrics coverage such as MCDC coverage.

The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.

Integration and development time

The final “primary” metrics are the development and integration time.  The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases).  The integration time is a bit more interesting.

In an ideal workflow for Model-Based Design, there is integrationHandsan integration model that is created at the start of the development cycle.  Individual developers check their models against that integration model to ensure that there are no conflicts.  Hence in an ideal flow, the integration time should be near zero.

However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes.  Like the bug detection finding integration issues is done further upstream in an MBD process.  Again the metric should use a weighted value based on the stage of where the integration issue is found.

Final thoughts

This post covered what can be measured, not how to measure them; this will be covered in future posts.  Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.


(1) For this post, I am assuming that you do not currently have metrics in place.  If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development.  Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process.  While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.

Model-Based Design foundational concepts

In previous blog posts, I have gone into some depth on testing, architecture and data management for models.  With this post, I will cover how these three activities for the foundation of any Model-Based Design process.

Foundations and core competencies

At later stages in the adoption of Model-Based Design crawl-walk-run-fly-300x191
processes task-specific groups will emerge (development, verification, and systems.)  However, at the start of the Model-BAsed Design process users from all groups need to determine the “common language” that will be used to develop their project (1).

Architecture and data sure, but why testing?

The identification of architecture and data as foundational concepts is generally well understood.modelcentricgifslow  Combined they define how people will develop the model through interfaces and clear communication.  So why testing in the trinity?  It returns to the core concept of Model-Based Design that models are at the center of the development process.  To ensure that the model can be used consistently through the development process they need to be “locked down”(2) with test cases.

Driven by this objective, the testing environment is designed at the start of the development process.  The requirements of the test environment should be addressed within the architectural and data infrastructure.  The good news is that best practices for the three “legs”(2) of the MBD stool are already in the well defined; it is a matter of honing them to your specific project and environment.

Final thoughts

This post is not intended to be technical; rather it is to remind us as we develop new processes to start out with the “best path forward” from the start.  In the section about the validation project, I will discuss the next round of tools that are commonly adopted.


(1) I may start using this graphic as my tag for MBD adoption.  Crawl = investigate.  Walk = initial.  Run = validated.  Flying = optimizing.
(2) Tests cases are elaborated as the model is developed.  The “lock down” is achieved through the use of a continuous build and test server.
(3) The metaphor of a tripod or stool can be overused.  But, to push it one last time, this is your stepping stool to the next round of MBD tools and processes.  Build it well and it forms a strong foundation.

Bug tracking software

The first thing I need to clarify in this post is, what is and what is not a bug.

A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
unintended ways.

A software bug is not is

  • Incomplete features:  During the software development process features will be under development(1).  As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
  • Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software.  Again, as long as the lack of the feature does not introduce errors it is not considered a bug.

Incomplete features should already be tracked in the project planning timeline.  Desired features should be incorporated into the requirements document for the next generation of the project.

Severity of bugs

Not all bugs are created equal; defining the sevBeetle-Poster-720x479erity of bugs is necessary for prioritizing the correction of the bugs.  There are two common metrics for determining severity.  Frequency, how often does the bug occur.  Impact, when the bug occurs what happens to the program.  The following lists provide examples of how frequency and impact could be defined(2).


  • Infrequent:  Happens in less than 1% of the executions of the program in the normal work tasks.
  • Common: Happens for 1 ~ 5% of users in the normal course of work.
  • Prevalent: Occurs for 10% of users in their normal course of work.


  • Low:  Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
  • Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
  • High: Bugs that crash the program or create a loss of data for the user.

Once the frequency and impact have been determined the ranking of the bug can be defined.

Low Medium High
Infrequent Advised  Recommended Required
Common Advised Required Required
Prevalent Recommended Required Required

The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).

Bug fixing workflow

There are multiple commercially available tools for bug tracking.  A basic workflow should include the following tasks and events.


  • Bug detection:  The bug is found either through use or captured in an existing test case
  • Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
  • Assignment:  The bug is assigned to a software engineer for resolution
  • Creation of test / validation of solution: If a test case does not already exist for the bug it should be created.  The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.

Final thoughts

Bug tracking and resolution is a problem common to all software development workflows.  The process for resolving these issues is the same for Model-Based Development as in traditional C development environments.  The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.


(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project.  See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development.  Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics.  Without the entry there is no method for determining the efficiency of the overall process.

What is a control algorithm?

The question, dreaded or loved that all engineers face is, “Tell me, what do you do?” As a controls engineer, I fall back to the following example.

Me: Imagine you are driving down the highway and you want to pass someone.  What do you do?
Imaginary person: Well I would shift over to the left lane and speed up.
Me: And if they started to go faster as you tried to pass them?
Imaginary person: Well I guess I would speed up some more.

Me: And after you pass them, what then?
Imaginary person:  Well I would move back over and slow back to my initial speed(1).

This simple example serves as a starting point for explaining the fundamentals of control algorithms.

  • The “objective”:  This is the thing we want to control.  In this example, it is the speed of our vehicle.passing-right
  • The input: How we affect the thing we want to control; for an automobile it is by depressing the acceleration pedal that the vehicle is commanded to go faster.
  • The feedback: The measurement of how close we are to our desired objective; e.g. are we going fast enough to pass the other car(2).

Everything is Newton’s method…

From the starting point of the automobile example, we can explain that the goal of a control algorithm is to drive the difference between the desired and the actual values to zero.  We can expand the driving example to drive our point home.newtons-method-example-graph

  • Overshoot:  When you stamp on the gas and end up going 80 instead of the 75 you needed to pass…
  • Fault detection: (Manual drivers only) when you try to hit 80 but are still in 2nd gear…
  • Adaptive controls: You drive more carefully in downtown Boston(3) than on a rural highway…

Final thoughts

As you could tell, this post is intended to be more light-hearted; however, I do find that thinking about controls problems in a non-technical relatable fashion helps me understand what I am talking about.

“It begins as an idea,
it ends with math(4)


(1) We know this is an imaginary person since they said they slowed back down after passing.
(2)In this example we actually have an indirect measurement, are we passing the car not the actual speed of the vehicle.
(3) At least I hope you allow more space between drivers when moving about in a rush hour situation.
(4) Frequently, in presentations, I will say “in the end, it all comes down to the math.”  While this is true, it lacks the motivation that drives the derivatives.