In this blog post, I will review my core V&V philosophy.  The base guidelines are common for both Model-Based Design and traditional textual based development workflows

Thou shall know what it is supposed to do…

V&V activities should be driven by the requirements for the product.  Without validating the requirements there is no assurance that the final deliverable will meet the customer’s needs.

Thou shall start testing at the start of development…

While not all tests can be run at the start of development the sooner test points can be locked down the less likely it is that issues will enter into the development.  The most common type of early testing include

1. Interface
2. Coverage
3. Range and overflow
4. Style guidelines

The advantage of performing coverage, range, and overflow testing at the start of development is that doing so simplifies these tests later on.  They can be viewed early on as preventive testing.

Thou shall develop reusable testing components…

As I have written about earlier in, ‘testing as software’ and ‘modular testing‘  development of reusable components is critical to the existence of a robust development environment.  The more you can reuse your test infrastructure the less likely that test based errors will exist.

Thou shall test to a tempo…

Your defined test suite should be run at a regular interval or on a predefined trigger. While not all tests will be run at each interval every test in the suite should be run at a minimum before version control branch is made.  Common intervals and triggers include

• Nightly:  these tend to be shorter faster running tests that provide “sanity” checks
• Weekly: these are the longer running tests that are often run over the weekend when processing power is freed up
• Check in: when an engineer checks in a file,  tests associated with that file are run
• Version control branch: prior to making a branch or revision point the full test suite should be run.  The branch or revision point should not be made until any issues are sorted out.

Thou shall make it easy to test…

If it is difficult for developers to run tests they won’t.  As a result, they may spend weeks developing only to find out they had a simple bug when they check software in.

Easy to test means 3 things

1. The developer does not need to know complex commands to execute the tests
2. The test should provide meaningful feedback to the developer
3. The test should run in a reasonable amount of time

Thou shall provide reports…

Test reports provide the information that the developers, system architects, and project planners use to track the progress of a project.  There should be a minimum of three types of reports

1. Detailed reports: These reports are read by the developer and provide direct feedback on the work they are doing
2. Summary reports: These reports are used by the system architect and the project planner.  They provide information on the current status of all the components
3. Trend reports: These reports provide information on the trends in tests, e.g. are the total number of issues increasing or decreasing over time.

Thou shall define passing…

For some tests there are clearly defined pass/fail criteria, my car either starts on a cold winters morning or it doesn’t.  For other tests, such as coverage or style guidelines thresholds are defined.

Thou shall automate…

The final commandment of V&V is to automate where it is possible.  Automation eliminates the possibility of human error and increases the likelihood that tests will be run with a regular tempo.

Final thoughts

These 8 commandments of V&V are things that most engineers know of as just common best practices.  The primary hurdles center around the initial implementation of the processes and the continued maintenance of them.  While there is an upfront and ongoing cost the overall cost savings from rugged V&V processes is significant.

It is in this are where MBD shines the most; by providing an infrastructure that is well suited to automation and “test as you grow” methodologies both large and small companies can implement effective V&V processes.