Modular testing environments

Foundations define and limit the structures we create; this is as true in Model-Based Design as it is in architecture.  With that in mind, I want to use this post to discuss the concept of modular testing environments (MTE).  First, I will point to an earlier blog post “Testing is software“, before I drill deeper into the concept of MTE.

What is a modular testing environment?

A modular testing environment consists of 5 parts

  1. Test manager:test manager provides the framework for running, evaluating and reporting on one or more test cases. Further, the test manager provides a single hook for the automation process.
  2. Test harnesses: a test harness is the software construct that “wraps” the unit under test.  Ideally, the test harness does not change the unit under test in any fashion; e.g. it allows ‘black box’ testing.
  3. Evaluation primitives: the evaluation primitives are a set of routines that are commonly used to evaluate the results of the test.  Evaluation primitives range from a simple comparison against an expected value to complex evaluations of a sequence of events.
  4. Reporting: there are two types of reports, human and machine readable.  The human readable reports are used as part of the qualification and review process.  Machine-readable reports are used for tracking of data across the project development.
  5. Data management: testing requires multiple types of data, inputs, outputs, parameters and expected results.

Why is a modular testing environment important?

Having helped hundreds of customers develop testing environments the 5 most common issues that I have encountered are

  1. Reinventing the wheel, wrong:  Even the simplest evaluation primitive can have unexpected complexities.  When people rewrite the same evaluation multiple times mistakes are bound to occur.
  2. Tell me what happened:  When tests are pulled together in an individual fashion it is common for there to be limited or inconsistent reporting methods.
  3. Fragile tests: A fragile test is one where if the inputs change in a significant fashion the test has to be completely rewritten.
  4. “Bob” has left the company:  Often tests are written by an individual and when that person leaves the information required to maintain those tests leaves with them.
  5. It takes too much time:  When engineers have to build up tests from scratch, versus assembling from components, it does take more time to create a test.  Hence, tests are not written.

Final thoughts

Verification and validation activities are central to any software development project, Model-Based Design or otherwise.  The easier you make the system to use the more your developers will embrace them.

Video Blog: Fault detection

This video blog looks at fault detection and error handling.  The included images of State Machines show templates for how I generally model fault detection algorithms.

In this first example there are two things to note:

  1. Debounce protection:  Returning from “move to fault” and “no fault.”  The signal needs to fall below the trigger signal – a delta to the signal.  This prevents “jitter” in the signal.  (Green circle.)
  2. Temporal logic:  The move to “in fault” only takes place after you have held the fault condition as true for a set period of time. (Orange circle and black circle.)


The next example is more complex; in this example, a single variable “engine temp” can result in two different error modes.  “High Temp” or “Critical High Temp.”  In reality, the pattern is a slight variation on the previous version however it shows how it can be expanded to more complex fault conditions.


User-friendly testing environments: Analysis and testing

Within a software development organization, whether for embedded code or a desktop application, there are distinct roles.  They are the controls engineer, the system architect, and the quality engineer.  Depending on the size of the development team some of these roles may be done by a single person.

Analysis versus testing

During the development phase of a project, the controls engineer should perform analysis tasks on the model.  These analysis tasks enable the controls engineer to determine if the algorithm they are developing is functionally correct and is compliance with the requirements.

It is common for the analysis tasks to be performed in an informal fashion.  It is common for engineers to simulate a model and then view the graphs of the outputs to determine if they have correctly implemented the algorithm.

The differentiating word in this description is informal.  When comparing analysis with testing we see that testing (either verification or validation) requires a formalized and “locked down” framework.  How then can the informal analysis be used during the formal testing?

Transitioning from analysis to testing

Ideally, the transition from informal analysis to functional combined-testing-analysistesting would flow seamlessly.   However, it is often the case that the work done in the analysis phase is thrown away in the transition to the testing phase.   This is understandable in a non-MBD environment but with the single truth approach of MBD, the analysis results should not be thrown away.  This is where the idea of “golden data” comes into use.  Golden Data is a set of data, both inputs, and outputs that an experienced engineer verifies as meeting the requirements of the algorithm.

Enabling the use of golden data to create test cases

who-moved-my-data-why-tracking-changes-and-sources-of-data-is-critical-to-your-data-lake-success-by-russ-savage-cask-3-638The easiest way to enable the use of golden data is to provide controls engineers with a simple interface in which they can provide the analysis data set and the information that transforms it into a testing data set.

Analysis data is transformed into test data by providing a method for “locking down” the results.  To locked down data the controls engineer needs to provide the test engineer information on what is expected from the analysis data.  This information could include the following types of golden data tests.

  • Strict:  The output data from testing must match the golden output data exactly.  This is normally done for Boolean or integer outputs.
  • Tolerance:  The output data from testing must match the golden output data within some bounded tolerance.  Tolerances can be absolute, or percentage.  Note special care needs to be taken with data with values around zero for percentage based tolerances.
  • Temporal: The output from testing must match the golden output data within some time duration.  These tests can also include tolerance and strict conditions.

In addition to the type of golden data tests to run the controls engineers should include information on which requirements the test maps onto.

Formal tests in support of development

In the same way that golden data can support testing the formal testing can support the controls engineers by informing them of the constraints that the requirements place on their design.  This can only be achieved if the tests are easy for the controls engineers to run.

What is “user-friendly?”

User-friendly interfaces for testing are defined by the following characteristics

  1. Data is accepted in “natural” format:  Any formatting or interpolation of the data is performed by the testing environment.
  2. Test results are presented in “human readable” format:  The results from the tests should be provided both in a summary format (pass/fail) and with detailed data, such as graphs and tabular data.
  3. Selection and execution of tests should be simple: Tests should be launchable from a user interface that provides a list of the valid tests and enables the running of tests in either single or batch modes.
  4. Test files should be automatically associated:  The management of test data (inputs and results) should be handled by the test manager.


Final thoughts

This blog post has described how information should be shared and how tests should be run.  In an upcoming post, I will cover the basics of modular test design.



Testing is software

This blog is a re-posting of early work from linkedin; I will be re-posting this week while I am at the Software Design for Medical Devices Europe conference in Munich.

Enabling the adoption of Model-Based Design

Test early, test often, test against requirements and test using formal methods. This is the mantra that developers (hopefully) hear. But what does it mean in practice? How do you produce effective and maintainable tests? I will argue that the first step is to think of test development in the same light as software development. Good testing infrastructure has requirements, is built from reusable components and written in a clear fashion to facilitate extensions and debugging efforts.

Why should you care?

In my 20+ years working in software, 2/3 of it in a consultative role, the most common problem I am called in to work on is mushroom code(1). Mushroom code is the end result of unstructured development, new algorithms are added on top of existing algorithms with little understanding of what it is feeding on. The result is an organic mess that is hard to sort out. This is prevalent in algorithmic development and even more common in testing which is often done “late and under the gun”

Testing components

A fully developed testing infrastructure consists of 5 components, a manager, execution methods, harnesses, reporting methods, and evaluation methods.

1.) Evaluation methods: use the data created through the execution of the test to determine the pass / fail / percentage complete status of the test:

Example a.) A MCDC test the evaluation would determine the percentage of conditions taken

Example b.) A regression test could compare the output values of between the baseline version of the code and the current release.

2.) Reporting methods: take the data from the evaluation methods and generate both human readable and history reports. The history reports are used to track overall trends in the software development process.(2)

3.) Harness: the harness provides a method for calling the unit under test (UUT) without modifying the UUT. Note test harnesses facilitate black box testing, e.g. the internal states of the unit under test are not known. However if internal states of the UUT are outputs at the root level of the model then white box testing can be done using the unit under test.(3)

4.) Execution methods: is how the test is run. This could be the simulation of a Simulink model, the execution of a .exe file, static testing (as with Polyspace) or the Real-Time execution (4)of the code.

As the name implies there is more than one “execution method.” They should be developed as a general class that allows the same method (simulation) to be applied to multiple harnesses. Each instance of a execution method applied to a harness is considered a test case.

5.) Test manager: is were all of these components come together. The test manager

  • Holds a list of the test cases
  • Automates the loading of associated test data
  • Triggers the execution of the test
  • Triggers the evaluation of the results
  • Triggers the generate of the test report

Sadly it will not yet fetch you a cold beverage.


1.) Mushroom code and spaghetti code are similar that they develop due to a lack of planning. Spaghetti code is characterized with convoluted calling structures; mushroom code is accumulation of code on top of code.

2.) An interesting list of what should go into a report can be found here.

3.) Any model can be turned into white box testing if global data is used. However the use of global data potential introduces additional failure cases.

4.) Yes, this blog retreads the work from 6 months ago, however it is good to review these issues.

Model-Based Design: Projects of interest

Early in my career one of my mentors made the statement

“If we understand the system we can model it.
If we can model it we can make predictions.
If we can make predictions we can make improvements”

In the past 20+ years, I have not heard a better statement of the driving ethos behind Model-Based Design.

If we understand the system, we can model it:(What we need to do): when the system is understood, it can be described mathematically.  This could be a derived first principal model or a statistical model; the important thing is that the confidence in the model fidelity is understood.

If we can model, we can make predictions:(What we can do): once the model is known it can be used.  The use of the model can be in the design of a controller, predicting a rainfall or embedded within a system to allow the system to respond with better insight.

If we can make predictions, we can make improvements:(Why we do it): this last part is the heart of Model-Based Design.  Once we can make accurate predictions we can use that information to improve what we are doing.

Model and equation…

Models build on a foundation of equationsflightdynamics to provide a dynamic, time variant representation of the real-world phenomenon.  Moreover those equations are working as part of a system; you leverage models when you move into complex systems with multiple interdependent equations.  Within the Model-Based Design world, we most often think of these systems as closed loop system.  Similar examples can be seen in the social sciences, in biology and chemistry.

Understanding from a sewage treatment plant…

Coming from an aerospace background, and starting my working career out in the automotive industry the general nature of models sunk in during one of my earliest consulting engagement; helping a customer model a sewage treatment plant to determine optimal processing steps against a set of formal requirements.

  • Requirements
    • The plant may not discharge more than N% of water in untreated state
    • The plant’s physical size cannot exceed Y square miles
  •  Objectives
    • Minimize the total processing cost of sewage treatment (weight: ω)
    • Minimize the total processing time of sewage (weight: λ)
    • Maximize the production of  energy from bio-gas (weight: Φ)
    • ….
  • Variants of inputs
    • Sewage inflow base rate has +/- 15% flow rate change
    • Extreme storm conditions can increase flow rate by 50%
    • ….

The final system model included bio-chemical reactions, fluid dynamic models, statistical “flush rates” and many other domains that I have now forgotten.  The final model was not able to answer all of the questions that the engineers had, however, it did allow them to design a plant with significantly lower untreated discharge rates and lower sewage processing costs.  This was possible because of the models.  This was the project that showed me just how expensive Model-Based Design is.