Foundations define and limit the structures we create; this is as true in Model-Based Design as it is in architecture.  With that in mind, I want to use this post to discuss the concept of modular testing environments (MTE).  First, I will point to an earlier blog post “Testing is software“, before I drill deeper into the concept of MTE.

What is a modular testing environment?

A modular testing environment consists of 5 parts

1. Test manager: a test manager provides the framework for running, evaluating and reporting on one or more test cases. Further, the test manager provides a single hook for the automation process.
2. Test harnesses: a test harness is the software construct that “wraps” the unit under test.  Ideally, the test harness does not change the unit under test in any fashion; e.g. it allows ‘black box’ testing.
3. Evaluation primitives: the evaluation primitives are a set of routines that are commonly used to evaluate the results of the test.  Evaluation primitives range from a simple comparison against an expected value to complex evaluations of a sequence of events.
4. Reporting: there are two types of reports, human and machine readable.  The human readable reports are used as part of the qualification and review process.  Machine-readable reports are used for tracking of data across the project development.
5. Data management: testing requires multiple types of data, inputs, outputs, parameters and expected results.

Why is a modular testing environment important?

Having helped hundreds of customers develop testing environments the 5 most common issues that I have encountered are

1. Reinventing the wheel, wrong:  Even the simplest evaluation primitive can have unexpected complexities.  When people rewrite the same evaluation multiple times mistakes are bound to occur.
2. Tell me what happened:  When tests are pulled together in an individual fashion it is common for there to be limited or inconsistent reporting methods.
3. Fragile tests: A fragile test is one where if the inputs change in a significant fashion the test has to be completely rewritten.
4. “Bob” has left the company:  Often tests are written by an individual and when that person leaves the information required to maintain those tests leaves with them.
5. It takes too much time:  When engineers have to build up tests from scratch, versus assembling from components, it does take more time to create a test.  Hence, tests are not written.

Final thoughts

Verification and validation activities are central to any software development project, Model-Based Design or otherwise.  The easier you make the system to use the more your developers will embrace them.