Foundations define and limit the structures we create; this is as true in Model-Based Design as it is in architecture. With that in mind, I want to use this post to discuss the concept of modular testing environments (MTE). First, I will point to an earlier blog post “Testing is software“, before I drill deeper into the concept of MTE.
What is a modular testing environment?
A modular testing environment consists of 5 parts
- Test manager: a test manager provides the framework for running, evaluating and reporting on one or more test cases. Further, the test manager provides a single hook for the automation process.
- Test harnesses: a test harness is the software construct that “wraps” the unit under test. Ideally, the test harness does not change the unit under test in any fashion; e.g. it allows ‘black box’ testing.
- Evaluation primitives: the evaluation primitives are a set of routines that are commonly used to evaluate the results of the test. Evaluation primitives range from a simple comparison against an expected value to complex evaluations of a sequence of events.
- Reporting: there are two types of reports, human and machine readable. The human readable reports are used as part of the qualification and review process. Machine-readable reports are used for tracking of data across the project development.
- Data management: testing requires multiple types of data, inputs, outputs, parameters and expected results.
Why is a modular testing environment important?
Having helped hundreds of customers develop testing environments the 5 most common issues that I have encountered are
- Reinventing the wheel, wrong: Even the simplest evaluation primitive can have unexpected complexities. When people rewrite the same evaluation multiple times mistakes are bound to occur.
- Tell me what happened: When tests are pulled together in an individual fashion it is common for there to be limited or inconsistent reporting methods.
- Fragile tests: A fragile test is one where if the inputs change in a significant fashion the test has to be completely rewritten.
- “Bob” has left the company: Often tests are written by an individual and when that person leaves the information required to maintain those tests leaves with them.
- It takes too much time: When engineers have to build up tests from scratch, versus assembling from components, it does take more time to create a test. Hence, tests are not written.
Verification and validation activities are central to any software development project, Model-Based Design or otherwise. The easier you make the system to use the more your developers will embrace them.